A N EQUIVALENCE BETWEEN DATA POISONING AND B YZANTINE GRADIENT ATTACKS
Anonymous authors
Paper under double-blind review
A BSTRACT
To study the resilience of distributed learning, the “Byzantine” literature considers a strong threat model where workers can report arbitrary gradients to the parameter server.
While this model helped generate several fundamental results, it has however sometimes been considered unrealistic, when the workers are mostly trustworthy machines.
In this paper, we show a surprising equivalence between this model and data poisoning, a threat considered much more realistic.
More speciﬁcally, we prove that any gradient attack can be reduced to data poisoning in a personalized federated learning system that provides PAC guarantees (which we show are both desirable and realistic in various personalized federated learning contexts such as linear regression and classiﬁcation).
Maybe most importantly, we derive a simple and practical attack that may be constructed against classical personalized federated learning models, and we show both theoretically and empirically the effectiveness of this attack.
I NTRODUCTION
Learning algorithms now typically leverage data generated by a large number of users (Smith et al., 2013; Wang et al., 2019a;b) to often learn a common model that ﬁts a large population (Konecn ´ y et al., 2015), but also sometimes to construct a personalized model for each individual (Ricci et al., 2011).
Autocompletion (Lehmann & Buschek, 2021), conversational (Shum et al., 2018) and recommendation (Ie et al., 2019) algorithms are examples of such algorithms deployed at scale.
To be effective, besides huge amounts of (distributed) data (Brown et al., 2020; Fedus et al., 2021), these algorithms require a high level of customization.
This has motivated research into personalized federated learning (Fallah et al., 2020; Hanzely et al., 2020; Dinh et al., 2020).
However, in applications such as content recommendation, activists, companies, and politicians have strong incentives to promote certain views, products or ideologies (Hoang, 2020; Hoang et al., 2021).
Remember for instance that, on YouTube, two views out of three result from algorithmic recommendations (Solsman, 2018).
Perhaps unsurprisingly, this has led to vast amounts of fabricated activities to bias algorithms (Bradshaw & Howard, 2019; Neudert et al., 2019), like “fake reviews” (Wu et al., 2020).
The scale of this phenomenon is well illustrated by the case of Facebook which, inalone, reported the removal of around 6 billion fake accounts from its platform (Fung & Garcia, 2019).
This is particularly concerning in the era of “stochastic parrots” (Bender et al., 2021): climate denialists are incentivized to pollute textual datasets with claims like “climate change is a hoax”, as autocompletion, conversational and recommendation algorithms trained on such data will more likely spread these views (McGufﬁe & Newhouse, 2020).
This raises serious concerns about the vulnerability of personalized federated learning to such misleading data.
Data poisoning attacks clearly constitute now a major machine learning security issue (Kumar et al., 2020).
Overall, in highly adversarial environments like social media, given the advent of deep fakes (Johnson & Diakopoulos, 2021), we should expect that most data are strategically crafted and labeled .
In this context, the authentication of the data provider seems critical.
In particular, the safety of learning algorithms arguably demands that they be trained solely on cryptographically signed data, that is, data that provably come from a known source.
But even signed data cannot be wholeheartedly trusted since users usually have preferences over what ought to be recommended to others.
They thus have incentives to behave strategically in order to promote certain views or products.
To address data poisoning, the Byzantine learning literature usually considers that each federated learning worker may behave arbitrarily (Blanchard et al., 2017; Mhamdi et al., 2018; El-Mhamdi et al., 2021).
Recall that at each iteration of the federated learning stochastic gradient descent, each worker is given the updated model, and is asked to compute the gradient of the loss function with respect to (a batch of) its local data.
Byzantine learning usually assumes that a malicious (Byzantine) worker may report any gradient; without having to justify whether such a gradient could have been generated through data poisoning.
In fact, the gradient attack threat model has sometimes been claimed to be unrealistic in practical federated learning (Shejwalkar et al., 2021), especially when the workers are machines owned by trusted entities (cross-silo FL (Kairouz et al., 2021)).
We prove in this paper an equivalence between gradients attacks and fabricated data injection, in a general and desirable collaborative learning framework.
Thereby, our paper provides the ﬁrst practically compelling argument for the necessity to protect federated learning against gradient attacks.
Contributions.
As a preamble of our main result, we formalize local PAC* learning 1 (Valiant, 1984) for personalized learning, and prove that a simple and general solution to personalized federated linear regression and classiﬁcation is indeed locally PAC* learning.
Our proof leverages a new concept called gradient-PAC* learning , which is of independent interest.
We prove that it is sufﬁcient to guarantee local PAC* learning, and that it is veriﬁed by basic learning algorithms, like linear and logistic regression.
This is an important and highly nontrivial contribution of this paper.
Our main contribution is then to prove that local PAC* learning in personalized federated learning essentially implies an equivalence between data poisoning and gradient attacks.
More precisely, we show how any (converging) gradient attack can be turned into a data poisoning attack, with the same resulting harm.
Given how easy it generally is to create fake accounts on web platforms and to inject data poisoning by generating fake activities, this result should arguably greatly increase the concerns over the vulnerabilities of federated learning with user-generated data.
Finally, we propose a simple but very general strategic gradient attack, called the counter-gradient attack (CGA), which any participant to federated learning can deploy to bias the global model towards any target model that better suits their interest.
We prove the effectiveness of this attack under fairly general assumptions, which apply to many proposed personalized learning frameworks including Hanzely et al.
Dinh et al.
We then show empirically how this gradient attack can be turned into a devastating data poisoning attack, with remarkably few data.
Related work.
Byzantine learning has provided both negative and positive results in Byzantine resilience (Blanchard et al., 2017; Mhamdi et al., 2018; Baruch et al., 2019; Xie et al., 2019), some of which apply almost straightforwardly to personalized federated learning (El-Mhamdi et al., 2020; 2021).
Such results study the resilience against a minority of adversarial users.
Our paper however focuses on a different kind of malicious users.
Namely, like Suya et al.
(2021), we study the resilience against strategic users, who aim to bias the learned models towards a speciﬁc target model.
The study of the resilience against strategic users is part of the research on strategyproof learning.
Many special cases of strategyproofness have been tackled, including regression (Chen et al., 2018b;
Dekel et al., 2010; Perote & Perote-Pe˜na, 2004; Ben-Porat & Tennenholtz, 2017), classiﬁcation (Meir et al., 2012; Chen et al., 2020; Meir et al., 2011; Hardt et al., 2016), statistical estimation (Cai et al., 2015), and clustering (Perote & Sevilla, 2003).
However, none of these papers tackles a general personalized federated learning scheme.
Typically, for linear regression, Chen et al.
(2018b)
and Perote & Perote-Pe˜na (2004) assume that each user can only provide a single data point.
This greatly restricts the users’ ability to contribute to the learning model.
And while Dekel et al.
(2010) allows multiple contributions, they either require payments, which might not be possible (e.g., due to ethical reasons), or they restrict the model to one dimension or a constant function in R d .
Conversely,
Suya et al.
(2021) show how to arbitrarily manipulate convex learning models through multiple data injections, in the case where a single model is learned from all data at once.
A large literature has focused on data poisoning , with either a focus on backdoor (Dai et al., 2019;
Zhao et al., 2020; Severi et al., 2021; Truong et al., 2020; Schwarzschild et al., 2021) or triggerless attacks (Biggio et al., 2012; Mu˜noz-Gonz´alez et al., 2017; Shafahi et al., 2018; Zhu et al., 2019;
Huang et al., 2020; Barreno et al., 2006; Aghakhani et al., 2021; Geiping et al., 2021).
However, most of this research analyzed data poisoning without signed data.
One noteworthy exception is
Mahloujifar et al.
(2019), whose universal attack ampliﬁes the probability of a (bad) property.
Collaborative PAC learning was previously introduced by Blum et al.
and then extensively studied (Chen et al., 2018a; Nguyen & Zakynthinou, 2018), sometimes with the presence of Byzantine collaborating users (Qiao, 2018; Jain & Orlitsky, 2020; Konstantinov et al., 2020).
We stress however that this line of work assumes that all honest users have the same labeling function.
In other words, given any query, they agree on how the query should be answered.
This is a very unrealistic assumption in many critical applications, like content moderation or language processing.
In fact, in such applications, removing outliers can be argued to amount to ignoring minorities’ views, which would be highly unethical.
The very deﬁnition of PAC learning must then be adapted, which is what we do in this paper (also, we adapt it to parameterized models).
Structure of the paper.
The rest of the paper is organized as follows.
Section 2 presents a general model of personalized learning, formalizes local PAC* learning and deﬁnes a general federated gradient descent algorithm.
Section 3 proves the equivalence between data poisoning and gradient attacks, under local PAC* learning.
Section 4 proves the local PAC* learning properties for federated linear regression and classiﬁcation.
Section 5 then describes a simple and general data poisoning attack, whose effectiveness against (cid:96) 22 is proved theoretically and empirically.
Section 6 concludes.
A GENERAL PERSONALIZED FEDERATED LEARNING FRAMEWORK
We consider a set [ N ] = { 1 , . . .
, N } of users.
Each user n ∈ [ N ] has a local signed dataset D n , and aims to learn a local model θ n ∈ R d .
Users may collaborate to improve their models.
Personalized learning must then input a tuple of users’ local datasets (cid:126) D (cid:44) ( D 1 , . .
, D N ) , and output a tuple of local models (cid:126)θ ∗ (cid:44) ( θ ∗ 1 , . .
. , θ ∗ N ) .
Like many others, we assume that the users perform federated learning to do so, by leveraging the computation of a common global model ρ ∈ R d .
Intuitively, the global model is an aggregate of all users’ local models, which users can leverage to improve their local models.
The common global model will typically allow users with too few data to obtain an effective local model, while it may be mostly discarded by users whose local datasets are large.
More formally, we consider a very general personalized learning framework which generalizes the models proposed by Dinh et al.
(2020) and Hanzely et al.
Namely, we consider that the personalized learning algorithm outputs a global minimum ( ρ ∗ , (cid:126)θ ∗ ) of a global loss given by
[Equation]
where R is a regularization, typically with a minimum at θ n = ρ .
For instance, Hanzely et al.
and Dinh et al.
deﬁne R ( ρ, θ n ) (cid:44) λ (cid:107) ρ − θ n (cid:107) 2 2 , which we shall call the (cid:96) 22 regularization.
But other regularizations may be considered, like the (cid:96) 2 regularization R ( ρ, θ n ) (cid:44)
λ (cid:107) ρ − θ n (cid:107) 2 , or the smooth- (cid:96) 2 regularization R ( ρ, θ n ) (cid:44) λ
(cid:113)
+ (cid:107) ρ − θ n (cid:107) 22 .
Note that, for all such regularizations, the limit λ → ∞ essentially yields the classical non-personalized federated learning framework.
L OCAL PAC* LEARNING
In this paper, we focus on personalized learning algorithms that provably recover a user n ’s preferred model θ † n , if the user provides a large enough honest dataset D n , i.e. constructed with θ † n .
Such honest datasets D n could typically be obtained by repeatedly drawing random queries (or features), and by using the user’s preferred model θ † n to provide (potentially noisy) answers (or labels).
We refer to Section 4 for examples.
The model recovery condition is then formalized as follows.
Deﬁnition 1.
A personalized learning algorithm is locally PAC
* learning if, for any subset H ⊂ [ N ] of honest users, any preferred models (cid:126)θ † , any ε, δ > 0 , and any datasets (cid:126) D − H from users n / ∈ H , there exists I such that, if all users h ∈ H provide honest datasets D h with at least |D h | ≥ I data points, then, with probability at least 1 − δ , we have (cid:13)(cid:13)(cid:13) θ ∗ h (cid:16) (cid:126) D (cid:17) − θ † h (cid:13)(cid:13)(cid:13) 2 ≤ ε for all users h ∈ H .
Local PAC* learning is arguably a very desirable property.
Indeed, it guarantees that any honest active user will not be discouraged to participate in federated learning as they will eventually learn their preferred model by providing more and more data.
In Section 4, we will show how local PAC* learning can be achieved in practice, by considering speciﬁc local loss functions L n .
2. F EDERATED GRADIENT DESCENT
While the computation of ρ ∗ and (cid:126)θ ∗ could be done by a single machine, which ﬁrst collects the datasets (cid:126) D and then minimizes the global loss L OSS , modern machine learning deployments often rather rely on federated (stochastic) gradient descent (or variants), with a central trusted parameter server.
In this setting, each user n keeps their data D n locally.
At each iteration t , the parameter server sends the latest global model ρ t to the users.
Each user n is then expected to update its local model given the global model ρ t , either by solving θ tn (cid:44) arg min θ n L n ( θ n , D n )+ R ( ρ t , θ n ) (which is what we assume in the theory part, in the manner of Dinh et al.
(2020)), or by making a (stochastic) gradient step from the previous local model θ t − 1 n (which is what is done in our experiments, in the manner of Hanzely & Richt´arik (2021)).
User n is then expected to report the gradient g tn = ∇ ρ R ( ρ t , θ tn ) to the parameter server.
The parameter server then updates the global model, using a gradient step, i.e. it computes ρ t +1 (cid:44) ρ t − η t (cid:80) n ∈ [ N ] g t n .
For simplicity, here, and since our goal is to show the vulnerability of personalized federated learning even in good conditions, we assume that the network is synchronous and that no node can crash.
Note also that our setting could be generalized to fully decentralized collaborative learning, as was done by El-Mhamdi et al.
We assume that the users are only allowed to send plausible gradient gradients.
More precisely, we denote G RAD ( ρ ) (cid:44) {∇ ρ R ( ρ, θ )
| θ ∈ R d } the closure set of plausible (sub)gradients at ρ .
If user n ’s gradient g tn is not in the set G RAD ( ρ t ) , the parameter server can easily detect the malicious behavior and g tn will be ignored by the parameter server at iteration t .
In the case of an (cid:96) 22 regularization, where R ( ρ, θ ) = λ (cid:107) ρ − θ (cid:107) 22 , we clearly have G RAD ( ρ )
= R d for all ρ ∈ R d .
It can be easily shown that, for (cid:96) 2 and smooth- (cid:96) 2 regularizations, G RAD ( ρ ) is the closed ball B (0 , λ ) .
Nevertheless, in this setting, a strategic user s ∈ [ N ] can deviate from its expected behavior, to bias the global model in their favor.
We identify in particular three sorts of attacks.
Data poisoning: Instead of collecting an honest dataset, s fabricates any strategically crafted dataset D s , and then performs all other operations as expected.
Model attack: At each iteration t , s ﬁxes θ ts (cid:44) θ ♠ s , where θ ♠ s is any strategically crafted model.
All other operations would then be executed as expected.
Gradient attack: At each iteration t , s sends any (plausible) strategically crafted gradient g ts .
Gradient attacks are intuitively the most harmful attacks, as the strategic user can then adapt their attack based on what they observe during the learning process.
However, because of this, gradient attacks are more likely to be ﬂagged as suspicious behaviors.
At the other end of the spectrum, data poisoning is a much safer attack, as the strategic user can always report their entire dataset, and prove that they rigorously performed the expected computations.
In fact, data poisoning can be executed, even if users directly provide the data to a (trusted) central authority, which is then tasked to execute (federated) gradient descent.
This is typically what is done to construct recommendation algorithms on social medias, where users’ data are their online activities (what they view, like and share).
Crucially, especially in applications with no clear ground truth, such as content moderation or language processing, the strategic user can always argue that their dataset is an “honest” dataset; not a strategically crafted one.
Ignoring the strategic user’s data on the basis that it is an “outlier” may then be regarded as unethical , as it can be argued to amount to rejecting minorities’ viewpoints.
T HE EQUIVALENCE BETWEEN DATA POISONING AND GRADIENT ATTACKS
We now present our main result.
The threat model we considered is closely related to “modeltargeted attacks”, studied in Suya et al.
Recall also that, in this theory part, at each iteration, local models θ tn are fully optimized, given ρ t , in the manner of Dinh et al.
Theorem 1 (Equivalence between gradient attacks and data poisoning) .
Assume local PAC* learning, and (cid:96) 2 2 or smooth- (cid:96) 2 regularization.
Assume also that L n is convex and L -smooth for all users n ∈ [ N ] , and that the learning rate η t is constant and small enough.
Consider any datasets (cid:126) D − s provided by users n (cid:54)
=
s .
Then, for any target model θ † s ∈ R d , there exists a converging gradient attack of strategic user s (i.e. g ts converges)
such that ρ t → θ † s , if and only if, for any ε > 0 , there exists a dataset D s such that (cid:13)(cid:13)(cid:13) ρ ∗ ( (cid:126) D ) − θ † s (cid:13)(cid:13)(cid:13) 2 ≤ ε .
Note that smoothness is used as a sufﬁcient condition to prove the convergence of federated gradient descent.
We now present our proof, which goes through the intermediary step of model attacks.
3. E QUIVALENCE BETWEEN DATA POISONING AND MODEL ATTACKS
In this section, we prove the equivalence between data poisoning and model attacks.
Lemma 1 (Reduction from model attack to data poisoning) .
Consider any data (cid:126) D and user s ∈ [ N ] .
Assume the global loss with datasets (cid:126) D has a global minimum ( ρ ∗ , (cid:126)θ ∗ ) .
Then ( ρ ∗ , (cid:126)θ ∗− s ) is also a global minimum of the modiﬁed loss with datasets (cid:126)
D − s and strategic reporting
θ ♠ s (cid:44) θ ∗ s ( (cid:126) D ) .
Proof.
The proof is almost straightforward.
It is given in Appendix B.1.
Now, intuitively, by virtue of local PAC* learning, strategic user s can essentially guarantee that the personalized learning framework will be learning θ ∗ s
≈ θ ♠ s .
But, a priori, it may seem unclear if this will imply a biasing of the global model essentially identical to the one obtained through the model attack that imposes θ ∗ s = θ ♠ s .
In the sequel, we show that the answer is yes.
Lemma 2 (Reduction from data poisoning to model attack) .
Assume (cid:96) 22 , (cid:96) 2 or smooth- (cid:96) 2 regularization, and assume local PAC* learning.
Consider any datasets D − s and any attack model
θ ♠ s such that the modiﬁed loss L OSS s has a unique minimum ρ ∗ ( θ ♠ s , (cid:126)
D − s ) , (cid:126)θ ∗− s ( θ ♠ s , (cid:126) D − s ) .
Then, for any ε > 0 , there exists a dataset D s such that we have
[Equation]
Our proof in fact holds for all continuous regularizations R with R ( ρ, θ ) → ∞ as (cid:107) ρ − θ (cid:107) 2 → ∞ .
Moreover, note that the approximation guarantee holds for local models too.
Sketch of proof.
Given local PAC* learning, for a large dataset D s constructed from θ ♠ s , strategic user s can guarantee θ ∗ s ( (cid:126) D )
≈ θ ♠ s .
By carefully bounding the effect of the approximation on the loss using the Heine-Cantor theorem, we show that this implies ρ ∗ ( (cid:126) D )
≈ ρ ∗ ( θ ♠ s , (cid:126)
D − s ) and θ ∗
n ( (cid:126) D )
≈ θ ∗ n ( θ ♠ s , (cid:126) D − s ) for all n (cid:54) =
s too.
The precise analysis, given in Appendix B.2, is nontrivial.
3. E QUIVALENCE BETWEEN MODEL ATTACKS AND GRADIENT ATTACKS
For the last few years, gradient attacks have been widely studied by the Byzantine learning literature.
Recently, Shejwalkar et al.
(2021) argued that they are not a realistic threat model.
Below, we prove that, in our setting, they are actually as concerning as model attacks (and, thus, as data poisoning).
Lemma 3 (Reduction from model attack to gradient attack) .
Assume that L n is convex and Lsmooth for all nodes n ∈ [ N ] , and that we use (cid:96) 22 or smooth- (cid:96) 2 regularization.
If g ts converges and if η t = η is a constant small enough, then ρ t will converge too.
Denote ρ ∞ its limit.
Then for any ε > 0 , there is θ ♠ s ∈ R d such that (cid:13)(cid:13)(cid:13)
[Equation]
Sketch of proof.
Denote g ∞ s the limit of g ts .
Gradient descent then behaves as though it was minimizing the loss plus ρ T g ∞ s (and ignoring R ( ρ, θ s ) ).
Essentially, classical gradient descent theory then guarantees the convergence of ρ t to ρ ∞ , though the precise proof is nontrivial.
Then, since G RAD is closed and g ∞ s ∈ G RAD , we can construct θ ♠ s which approximately yields the gradient g ∞ s .
The full proof (which also yields the necessary upper bound on η ) is given in Appendix B.3.
Since any model attack can clearly be achieved by the corresponding honest gradient attack, we conclude that model attacks and gradient attacks are essentially equivalent.
In light of our previous results, this implies that gradient attacks are essentially equivalent to data poisoning (Theorem 1).
E XAMPLES OF LOCALLY PAC* LEARNING SYSTEMS
To the best of our knowledge, though similar to collaborative PAC learning (Blum et al., 2017), local PAC* learnability is a new concept in the context of personalized federated learning.
It is thus important to show that it is not meaningless.
To achieve this, in this section, we provide sufﬁcient conditions for a personalized learning model to be locally PAC* learnable.
First, we construct local losses L n as sums of losses per input, i.e.
[Equation]
for some “loss per input” function (cid:96) and some weight ν > 0 .
Appendix C gives theoretical and empirical arguments are provided for using such a sum (as opposed to an expectation).
Remarkably, for linear or logistic regression, given such a loss, local PAC* learning can then be guaranteed.
Theorem 2 (Personalized least square linear regression is locally PAC* learning) .
Consider (cid:96) 22 , (cid:96)or smooth- (cid:96) 2 regularization.
Assume that, to generate a data x
i , a user with preferred parameter θ † ∈ R
d
ﬁrst independently draws a random vector query Q i ∈ R d from a sub-Gaussian query distribution ˜ Q , with parameter σ Q and positive deﬁnite matrix Σ
= E (cid:2) Q i Q Ti (cid:3) .
Assume that the user labels Q i with a real-valued answer A
i = Q T i θ † + ξ i , where ξ i is a zero-mean sub-Gaussian random noise with parameter σ ξ , independent from Q i and other data points.
Finally, assume that (cid:96) ( θ, Q i , A i ) =
12 ( θ T Q i −A i ) 2 .
Then the personalized learning algorithm is locally PAC* learning.
Theorem 3 (Personalized logistic regression is locally PAC*-learning) .
Consider (cid:96) 22 , (cid:96) 2 or smooth- (cid:96) 2 regularization.
Assume that, to generate a data x
i , a user with preferred parameter θ † ∈ R
d
ﬁrst independently draws a random vector query Q i ∈ R d from a query distribution ˜ Q , whose support S UPP ( ˜ Q ) is bounded and spans the full vector space R d .
Assume that the user then labels
Q i with answer A i = 1 with probability σ ( Q Ti θ † ) , and labels it
A i = − 1 otherwise, where σ ( z ) =
(1+ e − z ) − 1 is the sigmoid logistic function.
Finally, assume that (cid:96) ( θ, Q i , A i ) =
− ln( σ ( A i θ T Q i )) .
Then the personalized learning algorithm is locally PAC* learning.
4. P ROOF SKETCH
In this section, we outline the proofs of theorems 2 and 3, as they are interesting in their own sake.
The proofs both leverage the following notion, which intuitively means “robust PAC* learning”.
Deﬁnition 2 (Gradient-PAC*) .
Denote E ( D , θ † , I , A, B, α ) the event ∀
[Equation]
The loss (cid:96) is gradient-PAC
* if, for any K > 0 , there exist constants A K , B K > 0 and α K < 1 , such that for any preferred model θ † ∈ R d with (cid:13)(cid:13) θ † (cid:13)(cid:13) 2 ≤ K , assuming that the dataset D is obtained by honestly collecting and labeling I data points according to the preferred model θ † , the probability of the event E ( D , θ † , I , A K , B K , α K ) goes to 1 as I → ∞ .
Intuitively, this deﬁnition asserts that, as we collect more data from an honest user, then, with high probability, the gradient of the loss at any point θ too far from θ † will point away from θ † .
In particular, gradient descent is then essentially guaranteed to draw θ closer to θ † .
The right-hand side of equation 4 is subtly chosen to be strong enough to yield local PAC* learning guarantees, and weak enough to be veriﬁed by linear and logistic regression, as proved by the following lemma.
Lemma 4.
Logistic and linear regression, deﬁned in theorems 2 and 3, are gradient PAC* learning.
Sketch of proof.
In the case of linear regression, remarkably, the discrepancy between the empirical and the expected loss functions depends only on a few key random variables, such as min S P (cid:16)
I (cid:80) Q i Q Ti (cid:17) and (cid:80) ξ i Q i , which can be controlled by appropriate concentration bounds.
Meanwhile, for logistic regression, for | b | ≤ K , we observe that ( a − b )( σ ( a ) − σ ( b )) ≥ c K min( | a − b | , | a − b | 2 ) .
Essentially, this proves that gradient-PAC* would hold if the empirical loss was replaced by the expected loss (times I ).
The actual proofs, however, are nontrivial, especially in the case of logistic regression, which leverages topological considerations to derive a critical uniform concentration bound.
The full proofs are given in appendices D.2 and D.3.
Now, under very mild assumptions on the regularization R (not even convexity!), which are veriﬁed by the (cid:96) 22 , (cid:96) 2 and smooth- (cid:96) 2 regularizations, we prove that the gradient-PAC* learnability through (cid:96) sufﬁces to guarantee that personalized learning will be locally PAC* learning.
Lemma 5.
Assume R is (sub)differentiable and nonnegative, and R ( ρ, θ ) → ∞ as (cid:107) ρ − θ (cid:107) 2 → ∞ .
If (cid:96) is gradient-PAC* and nonnegative, then personalized learning is locally PAC*-learning.
Sketch of proof.
Once other users’ datasets are ﬁxed, the learning of an honest user’s model has a ﬁxed biased due to R .
But as the user provides more data, by gradient-PAC*, the local loss becomes predominant, which guarantees local PAC*-learning.
Appendix E provides a full proof.
Combining the two lemmas clearly yields theorems 2 and 3 as special cases.
Note that our result actually applies to a more general set of regularizations and a more general set of local losses.
4. T HE CASE OF NEURAL NETWORKS
Neural networks generally do not verify gradient PAC* learning.
After all, because of symmetries like neuron swapping, different values of the parameters compute the same neural network function.
Thus the notion of a “preferred model” θ † is arguably ill-deﬁned for neural networks 2 .
Nevertheless, we may consider a strategic user who only aims to bias the parameters of the last layer.
In particular, assuming that all layers but the last one of a neural network are pretrained and ﬁxed, then our theory may apply to the parameters of the last layer, if it performs classiﬁcation or regression.
A PRACTICAL DATA POISONING ATTACK
We now construct a practical data poisoning attack.
We do so by introducing a new effective gradient attack, and by then leveraging our equivalence to turn it into a data poisoning attack.
5. T HE COUNTER - GRADIENT ATTACK
We now deﬁne a simple, general and practical gradient attack, which we call the counter-gradient attack (CGA).
Intuitively, this attack estimates the sum g † ,t − s of the gradients of other users based on its value at the previous iteration, which can be inferred from the way the global model ρ t − 1 was updated into ρ t .
More precisely, apart from initialization ˆ g 1 − s (cid:44) 0 , CGA makes the estimation
[Equation]
Strategic user s then reports the plausible gradient that moves the global model closest to the user’s target model θ † s , assuming others report ˆ g t − s .
In other words, at every iteration, CGA reports
[Equation]
Note that, to perform this attack, user s only needs to know the previous and current learning rates η t − 1 and η t , the previous and current global models ρ t − 1 and ρ t , and their target model θ † s .
Computation of CGA.
Deﬁne h ts (cid:44) g
[Equation]
1 s +
[Equation]
− ρ t − 1 − ρ t η t − 1 .
For convex sets G RAD ( ρ t ) , it is straightforward to see that CGA boils down to computing the orthogonal projection of h ts on
G RAD ( ρ t ) .
This yields very simple computations for (cid:96) 22 , (cid:96) 2 and smooth- (cid:96) 2 regularizations.
Proposition 1.
For (cid:96) 22 regularization, CGA reports g ts = h ts .
For (cid:96) 2 or smooth- (cid:96) 2 regularization,
CGA reports g t s = h t s min { 1 , λ/ (cid:107) h t s (cid:107) 2 } .
Proof.
Equation (6) boils down to minimizing the distance between ρ t − θ † s η t − ˆ g t − s and G RAD ( ρ ) , which is the (convex) ball B (0 , λ ) .
This minimum is the orthogonal projection on B (0 , λ ) .
Theoretical analysis.
We now prove that CGA is perfectly successful against (cid:96) 22 regularization.
To do so, we suppose that, at each iteration t and for each user n (cid:54) = s , the local models θ n are fully optimized with respect to ρ t , and the honest gradients of g † ,tn are used for the gradient descent of ρ .
Theorem 4.
Consider (cid:96) 22 regularization.
Assume that (cid:96) is convex and L (cid:96) -smooth, and that η t = η is small enough.
Then CGA is converging and optimal, as ρ t → θ † s .
Sketch of proof.
The main challenge is to guarantee that the other users’ gradients g † ,tn for n (cid:54) =
s remain sufﬁciently stable over time to guarantee convergence, which can be done by leveraging
L -smoothness.
The full proof, with the necessary upper-bound on η , is given in Appendix F.
The analysis of the convergence against smooth- (cid:96) 2 is unfortunately signiﬁcantly more challenging.
Here, we simply make a remark about what CGA yields in this case, if it converges.
Proposition 2.
If CGA against smooth- (cid:96) 2 regularization converges for η t = η , then it either achieves perfect manipulation, or it is eventually partially honest, in the sense that the gradient by CGA correctly points towards θ † s .
Proof.
Denote P the projection onto the closed ball B (0 , λ ) .
If CGA converges, then, by Proposition 1, P (cid:16) g ∞ s + ρ ∞ − θ † s η (cid:17) =
g ∞ s .
This implies in particular that ρ ∞ − θ † s and g ∞ s must be colinear.
If perfect manipulation is not achieved (i.e. ρ ∞ (cid:54) = θ † s ), then we must have g ∞ s = λ ρ ∞ − θ
[Equation]
s (cid:107) ρ
[Equation]
s (cid:107)
[Figure]
Empirical evaluation of CGA.
We deployed CGA to bias the federated learning of MNIST.
We consider a strategic user whose target model is one that labels 0 ’s as 1 ’s, 1 ’s as 2 ’s, and so on, until 9 ’s that are labeled as 0 ’s. In particular, this target model has a nil accuracy.
Figure 1 shows that such a user effectively hacks the (cid:96) 22 regularization against 10 honest users who each have 6,000 data points of MNIST, in the case where local models only undergo a single gradient step at each iteration, but fails to hack the (cid:96) 2 regularization.
Further details on the experiment are given in Appendix G. We also ran a similar successful attack on the last layer of a deep neural network trained on cifar-10, which is detailed in Appendix J. The Appendix also discusses the extent to which the attack may be turned into data poisoning.
5. F ROM GRADIENT ATTACK TO MODEL ATTACK AGAINST (cid:96)
We now show how to turn a gradient attack into model attack, against (cid:96) 22 regularization.
Assume that we found a gradient g ∞ s such that ρ ∞ = θ † s .
It is trivial to transform it into a model attack by setting θ ♠ s (cid:44) θ † s − 12 g ∞ s , as guaranteed by the following result, and as depicted by Figure 2.
[Figure] [Figure]
Accuracy of ρ t according to θ † s (which relabels
[Table]
Proposition 3.
Consider the (cid:96) 22 regularization.
Suppose that g ts → g ∞ s and ρ t → θ † s , with a constant learning rate η t = η .
Then, under the model attack θ ♠ s (cid:44) θ † s − 12 λ g ∞ s , the gradient at ρ = θ † s vanishes, i.e. ∇ ρ L OSS ( θ † s , (cid:126)θ ∗− s ( θ † s , (cid:126) D − s ) , θ ♠ s , D − s ) = 0 .
Proof.
Given that the learning rate is constant, the convergence ρ t → θ † s implies that the sum of honest users’ gradients at ρ =
θ † s must equal − g ∞ s .
Therefore, to achieve ρ ∗ = θ † s with a model attack, it sufﬁces to send a model θ
♠ s such that the gradient of λ (cid:13)(cid:13) ρ − θ ♠ s (cid:13)(cid:13) 2 2 with respect to ρ at ρ = θ † s equals g ∞ s .
Since the gradient is λ ( θ † s − θ ♠ s ) , θ ♠ s (cid:44) θ † s − 12 λ g ∞ s does the trick.
5. F ROM MODEL ATTACK TO DATA POISONING AGAINST (cid:96)
The case of linear regression.
In linear regression, any model attack can be turned into a single data poisoning attack, as proved by the following theorem whose proof is given in Appendix H.
Theorem 5.
Consider the (cid:96) 22 regularization and linear regression.
For any data D − s and any target value θ † s , there is a datapoint ( Q , A ) to be injected by user s such that ρ ∗ ( { ( Q , A ) } , D − s )
= θ † s .
The case of linear classiﬁcation.
We now consider linear classiﬁcation, with the case of MNIST.
By Theorem 2, any model attack can be turned into data poisoning, provided sufﬁciently many (random) data points are labeled by the strategic user.
However, this may require creating too many data labelings, especially if the norm of θ ♠ s is large (which is the case if s is alone against many active users), as suggested by Theorem 3.
For efﬁcient data poisoning, deﬁne the indifference afﬁne subspace V ⊂ R d as the set of images with equiprobable labels.
Intuitively, labeling images close to V is very informative, as it informs us directly about the separating hyperplanes.
To generate images, we ﬁrst draw random images, which we then project orthogonally on V .
We then add a small noise, before probabilistically labeling the image with model
θ ♠ s .
Note that this leads us to consider images not in [0 , 1] d .
Figure 3 shows the effectiveness of the resulting data poisoning attack, with only 2,000 data points, as opposed to the other nodes’ 6,000 data points.
More details and explanations are provided in Appendix I.
[Figure]
C ONCLUDING R EMARKS
We showed in this paper that, unlike what has been argued by, e.g., Shejwalkar et al.
(2021), the gradient attack threat is not unrealistic.
For personalized federated learning with local PAC* guarantees, gradient attacks are in fact just as realistic and harmful as strategic data reporting.
More generally, our work stresses how critical Byzantine learning research is.
For instance, El-Mhamdi et al.
(2021) proved lower bounds on what any collaborative learning algorithm can guarantee in heterogeneous environments, under Byzantine gradient attacks.
Our work implies that, at least for certain personalized federated learning problems, these lower bounds also hold for data poisoning attacks, which are known to be common for many high-risk applications.
Arguably, a lot more security measures are urgently needed to make large-scale learning algorithms safe.
E THICS STATEMENT
The safety of algorithms is arguably a prerequisite to their ethics.
After all, an arbitrarily manipulable large-scale algorithm will unavoidably endanger the targets of the entities that successfully design such algorithms.
Typically, unsafe large-scale recommendation algorithms may be hacked by health disinformation campaigns that aim to promote non-certiﬁed products, e.g., by falsely pretending that they cure COVID-19.
Such algorithms must not be regarded as ethical, even if they were designed with the best intentions.
We believe that our work helps understand the vulnerabilities of such algorithms, and will motivate further research in the ethics and security of machine learning.
R EPRODUCIBILITY STATEMENT
All our experiments are run on the classical datasets MNIST and FashionMNIST.
We provide all of the source codes to reproduce the experiments:
• The sum versus expectation experiments can be run by executing this ﬁle:
https://www.dropbox.com/sh/qdgmz9air24nhyr/AAAtycEkxc_ 1hGbvU5YG18z4a?dl= •
The counter-gradient attack experiments can be run by executing this ﬁle:
https://www.dropbox.com/sh/bycqkccgmk4muzn/ AACRD1yeTglLSHEd1OOAzmVqa?dl= • The data poisoning attack experiments can be run by executing this ﬁle:
https://www.dropbox.com/sh/qodnl6ivzti8hch/ AADgX4EYuSOotiMCAHyTIiGMa?dl= • The cifar10 on VGG 13-BN experiments can be run by executing this ﬁle:
https://www.dropbox.com/sh/5mw5c9dt25eiav7/AAC0jP3e2kuh_ zvLudWbmDl-a?dl=
The experiments are seeded and the CuDNN backend is conﬁgured in deterministic mode in order to reduce the sources of non-determinism.
We also turn of the benchmark mode.
Executing the codes will generate the ﬁgures and statistics of our main paper, and most of the ﬁgures of our Appendix.
Our other ﬁgures can be obtained by adjusting the hyperparameters of our codes.
The full description of the architecture and optimisation algorithm used is described in Appendix C.
The experimental setup details of each experiment are provided in the Appendix, along with additional results.
The Appendix also contains the full proofs of our theorems.
Appendices A C ONVEXITY LEMMAS A. G ENERAL LEMMAS
Deﬁnition 3.
We say that f : R d → R is locally strongly convex if, for any convex compact set
C ⊂ R d , there exists µ > 0 such that f is µ -strongly convex on C , i.e. for any x, y ∈ C and any λ ∈ [0 , 1] , we have
[Equation]
It is well-known that if f is differentiable, this condition amounts to saying that (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) 2 ≥ µ (cid:107) x − y (cid:107) 2 for all x, y ∈ C .
And if f is twice differentiable, then it amounts to saying ∇ 2 f ( x ) (cid:23) µI for all x ∈ C .
Lemma 6.
If f is locally strongly convex and g is convex, then f
+ g is locally strongly convex.
Proof.
Indeed, ( f + g )( λx + (1 − λ ) y ) ≤ λf ( x ) + (1 − λ )
f ( y ) − µ 2 λ (1 − λ ) (cid:107) x − y (cid:107)
+ λg ( x ) + (1 − λ ) g
( y ) =
λ ( f + g )( x ) + (1 − λ )( f + g )( y ) − µ 2 λ (1 − λ ) (cid:107) x − y (cid:107) 22 .
Deﬁnition 4.
We say that f : R d → R is L -smooth if it is differentiable and if its gradient is
L -Lipschitz continuous, i.e. for any x, y ∈ R d ,
[Equation]
(cid:107) 2 .
Lemma 7.
If f is L f -smooth and g is L g -smooth, then f + g is ( L f + L g ) -smooth.
Proof.
Indeed, (cid:107)∇ ( f + g )( x ) − ∇ ( f + g )( y ) (cid:107) 2 ≤ (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) 2 + (cid:107)∇ g ( x ) − ∇ g ( y ) (cid:107) 2 ≤
L
[Equation]
Lemma 8.
Suppose that f : R d × R d (cid:48) (cid:55)→ R is locally strongly convex and L -smooth, and that, for any x ∈ X , where X ⊂ R d is a convex compact subset, the map y
(cid:55)→ f ( x, y ) has a minimum y ∗ ( x ) .
Note that local strong convexity guarantees the uniqueness of this minimum.
Then, there exists K such that the function y ∗ is K -Lipschitz continuous on X .
Proof.
The existence and uniqueness of y ∗ ( x ) hold by strong convexity.
Fix x, x (cid:48) .
By optimality of y ∗ , we know that ∇ y f
( x, y ∗ ( x ))
=
∇ y f
( x (cid:48) , y ∗ ( x (cid:48) ))
= 0 .
We then have the following bounds
[Equation] [Equation]
where we ﬁrst used the local strong convexity assumption, then the fact that ∇ y f
( x, y ∗ ( x ))
= 0 , then the fact that ∇ y f
( x (cid:48) , y ∗ ( x (cid:48) ))
= 0 , and then the L -smooth assumption.
Lemma 9.
Suppose that f : R d × R d (cid:48) (cid:55)→ R is locally strongly convex and L -smooth, and that, for any x ∈ X , where X ⊂ R d is a convex compact subset, the map y
(cid:55)→ f ( x, y ) has a minimum y ∗ ( x ) .
Deﬁne g
x ) (cid:44) min
y ∈
Y f
( x, y ) .
Then g is convex and differentiable on X and ∇ g ( x ) =
∇ x f ( x, y ∗ ( x )) .
Proof.
First we prove that g is convex.
Let x 1 , x 2 ∈ R d , and λ 1 , λ 2 ∈ [0 , 1] with λ 1 + λ 2 = 1 .
For any y 1 , y 2 ∈ R d (cid:48) , we have
[Equation] [Equation]
, y 2 ) .
, which proves the
Now denote h ( x ) = ∇ x f ( x, y ∗ ( x )) .
We aim to show
that ∇ g ( x ) =
h ( x ) .
Let ε ∈ R d small enough so that x + ε ∈ X .
Now note that we have
[Equation]
which shows that h ( x ) is a superderivative of g at x .
We now show that it is also a subderivative.
To do so, ﬁrst note that its value at x + ε is approximately the same, i.e.
[Equation]
where we used the L -smoothness of f and Lemma 8.
Now notice that
[Equation]
But we know that (cid:107) h
( x + ε )
− h
( x ) (cid:107)
= O
[Equation]
Rearranging the terms then yields
[Equation]
which shows that h ( x ) is also a subderivative.
Therefore, we know that g ( x + ε ) =
g ( x )+ ε T h ( x )+ o ( (cid:107) ε (cid:107) 2 ) , which boils down to saying that g is differentiable in x ∈ X , and that ∇ g ( x ) = h ( x ) .
Lemma 10.
Suppose that f : X × R d (cid:48)
→ R is µ -strongly convex, where X ⊂ R d is closed and convex.
Then g : X → R , deﬁned by g ( x ) =
inf
y ∈ Y f
( x, y ) , is well-deﬁned and µ -strongly convex too.
Proof.
The function y (cid:55)→ f
( x, y ) is still strongly convex, which means that it is at least equal to a quadratic approximation around 0, which is a function that goes to inﬁnity in all directions as (cid:107) y (cid:107) 2 → ∞ .
This proves that the inﬁmum must be reached within a compact set, which implies the existence of a minimum.
Thus g is well-deﬁned.
Moreover, for any x 1 , x 2 ∈ X, y 1 , y 2 ∈ R d (cid:48) , and λ 1 , λ 2 ≥ 0 with λ 1 + λ 2 = 1 , we have
[Equation]
where we used the µ -strong convexity of f .
Taking the inﬁmum over y 1 , y 2 implies the µ -strong convexity of g .
A. A PPLICATIONS TO L OSS
Now instead of proving our theorems for different cases separately, we make the following assumptions on the components of the global loss that encompasses both (cid:96) 22 and smooth- (cid:96) 2 regularization, a well as linear regression and logistic regression.
Assumption 1.
Assume that (cid:96) is convex and L (cid:96) -smooth, and that R ( ρ, θ )
= R 0 ( ρ − θ ) , where
R 0 : R d → R is locally strongly convex (i.e. strongly convex on any convex compact set), L R 0smooth and satisfy R 0 ( z ) =
Ω( (cid:107) z (cid:107) 2 ) as (cid:107) z (cid:107) 2 → ∞ .
Lemma 11.
Under Assumption 1, L OSS is locally strongly convex and L -smooth.
Proof.
All terms of L OSS are L 0 -smooth, for an appropriate value of L 0 .
By Lemma 7, their sum is thus also L -smooth, for an appropriate value of L .
Now, given Lemma 6, to prove that L OSS is locally strongly convex, it sufﬁces to prove that ν (cid:80) (cid:107) θ n (cid:107) 22 + R 0 ( ρ − θ 1 ) is locally strongly convex.
Consider any convex compact set C ⊂ R d × (1+ N ) .
Since R 0 is locally strongly convex, we know that there exists µ > 0 such that ∇ 2 R 0 (cid:23) µI .
As a result,
[Equation]
Now deﬁne α (cid:44) (cid:113) 2 µ ν +2 µ .
Clearly, 0 < α < 1 .
Moreover, 0 ≤ (cid:13)(cid:13) 1 α θ 1 − αρ (cid:13)(cid:13) 2 2 = 1 α 2 (cid:107) θ 1 (cid:107)
+ α 2 (cid:107) ρ (cid:107) 22 − 2 ρ T θ 1 .
Therefore 2 ρ T θ 1 ≤ α 2 (cid:107) ρ (cid:107) 22 + 1 α 2 (cid:107) θ 1 (cid:107) 22 , which thus implies ( ρ
[Equation]
[Table]
[Equation]
[Table]
[Equation]
which proves that ∇ 2 L OSS (cid:23) κI , with κ > 0 .
This shows that L OSS is locally strongly convex.
Lemma 12.
Under Assumption 1, ρ (cid:55)→ (cid:126)θ ∗ ( ρ, (cid:126) D ) is Lipchitz continuous on any compact set.
Proof.
Deﬁne f n ( ρ, θ n ) (cid:44) ν
(cid:107) θ n (cid:107) 22 + (cid:80) x ∈D n (cid:96) ( θ n , x ) + λ (cid:107) ρ − θ n (cid:107) 22 .
If (cid:96) is L -smooth, then f n is clearly ( |D n | L + ν + λ ) -smooth.
Moreover, if (cid:96) is convex, then for any ρ , the function θ n (cid:55)→ f n ( ρ, θ n ) is at least ν -strongly convex.
Thus Lemma 8 applies, which guarantees that ρ (cid:55)→ (cid:126)θ ∗ ( ρ, (cid:126) D ) is Lipchitz.
Lemma 13.
Under Assumption 1, ρ (cid:55)→ L OSS ( ρ, (cid:126)θ ∗ ( ρ, (cid:126) D ) , (cid:126) D ) is L -smooth and locally strongly convex.
Proof.
By Lemma 11, the global loss is known to be L -smooth, for some value of L and locally strongly convex.
Denoting f : ρ (cid:55)→ L OSS ( ρ, (cid:126)θ ∗ ( ρ, (cid:126) D ) , (cid:126) D ) , we then have
[Equation]
[Equation] [Equation]
which proves that f is L -smooth.
For strong convexity, note that since the global loss function is locally strongly convex, for any compact convex set C , there exists µ such that L OSS ( ρ, (cid:126)θ, (cid:126) D ) is µ -strongly convex on C
=
( C 1 , C 2 )
⊂ ( R d , R N × d ) , therefore, by Lemma 10, f ( ρ ) will also be µ -strongly convex on C 1 which means that f ( ρ ) is locally strongly convex.
B P ROOF OF THE EQUIVALENCE B. P ROOF OF THE REDUCTION FROM MODEL ATTACK TO DATA POISONING
Proof of Lemma 1.
We omit making the dependence of the optima on (cid:126) D explicit, and we consider any other models ρ and (cid:126)θ − s .
We have the following inequalities:
[Equation] [Equation]
[Equation]
where we used the optimality of ( ρ ∗ , (cid:126)θ ∗ ) in the second line, and where we repeatedly used the fact that θ ∗ s = θ ♠ s .
This proves that ( ρ ∗ , (cid:126)θ ∗− s ) is a global minimum of the modiﬁed loss.
B. P ROOF OF THE REDUCTION FROM DATA POISONING TO MODEL ATTACK
First, we deﬁne the following modiﬁed loss function:
[Equation] [Equation]
where (cid:126)θ − s and (cid:126) D − s are variables and datasets for users n (cid:54)
=
s .
We then deﬁne ρ ∗ ( θ ♠ s , (cid:126) D − s ) and (cid:126)θ ∗− s ( θ ♠ s , (cid:126)
D − s ) as a minimum of the modiﬁed loss function, and θ ∗ s ( θ ♠ s , (cid:126) D − s ) (cid:44) θ ♠ s .
We now prove a slightly more general version of Lemma 2, which applies to a larger class of regularizations.
It also shows how to construct the strategic’s user data poisoning attack.
Lemma 14 (Reduction from data poisoning to model attack) .
Assume local PAC* learning.
Suppose also that R is continuous and that R ( ρ, θ ) → ∞ when (cid:107) ρ − θ (cid:107) 2 → ∞ .
Consider any datasets D − s and any attack model
θ ♠ s such that the modiﬁed loss L OSS s has a unique minimum ρ ∗ ( θ ♠ s , (cid:126)
D − s ) , (cid:126)θ ∗− s ( θ ♠ s , (cid:126) D − s ) .
Then, for any ε, δ > 0 , there exists I such that if user s ’s dataset D s contains at least I inputs drawn from model
θ ♠ s , then, with probability at least 1 − δ , we have
[Equation]
Clearly, (cid:96) 22 , (cid:96) 2 and smooth- (cid:96) 2 are continuous regularizations, and verify R ( ρ, θ ) → ∞ when (cid:107) ρ − θ (cid:107) 2 → ∞ .
Moreover, setting δ (cid:44) 1 / 2 shows that the probability that the dataset D s satisﬁes the inequalities of Lemma 14 is positive.
This implies in particular that there must be a dataset
D s that satisﬁes these inequalities.
ALl in all, this shows that Lemma 14 implies Lemma 2.
Proof of Lemma 14.
Let ε, δ >
and θ ♠ s ∈ R d .
Denote ρ ♠ (cid:44) ρ ∗ ( θ ♠ s , (cid:126) D − s ) and (cid:126)θ ♠ (cid:44) (cid:126)θ ∗ ( θ ♠ s , (cid:126)
D − s )
the result of strategic user s ’s model attack.
We deﬁne the compact set C by
[Equation]
We deﬁne D (cid:44) R d × N − C
the closure of the complement of C .
Clearly, ρ ♠ , (cid:126)θ ♠− s / ∈ D .
We aim to show that, when strategic user s reveals a large dataset D s whose answers are provided using the attack model θ ♠ s , then the same holds for any global minimum of the global loss ρ ∗ ( (cid:126) D ) , (cid:126)θ ∗− s ( (cid:126) D ) ∈
C .
Note that, to prove this, it sufﬁces to prove that the modiﬁed loss takes too large values, even when θ ♠ s is replaced by θ ∗ s ( (cid:126) D ) .
Let us now formalize this.
Denote L ♠ (cid:44) L OSS s ( ρ ♠ , (cid:126)θ ♠− s , θ ♠ s , (cid:126) D − s ) .
We deﬁne
[Equation] [Equation]
By a similar argument as that of Lemma 5, using the assumption R → ∞ at inﬁnity, we know that the inﬁmum is actually a minimum.
Moreover, given that the minimum of the modiﬁed loss L OSS s is unique, we know that the value of the loss function at this minimum is different from its value at ρ ♠ , (cid:126)θ ♠− s .
As a result, we must have η > 0 .
Now, since the function R is differentiable, it must be continuous.
By the Heine–Cantor theorem, it is thus uniformly continuous on all compact sets.
Thus, there must exist κ > 0 such that, for all models θ s satisfying (cid:13)(cid:13) θ s − θ ♠ s (cid:13)(cid:13) 2 ≤ κ , we have
[Equation]
Now, Lemma 5 guarantees the existence of I such that, if user s provides a dataset D s of least I answers with the model
θ ♠ s , then with probability at least 1 − δ , we will have (cid:13)(cid:13)(cid:13) θ ∗ s ( (cid:126) D ) − θ ♠ s (cid:13)(cid:13)(cid:13) 2 ≤ min( κ, ε ) .
Under this event, we then have
[Equation]
Then
[Equation]
This shows that there is a high probability event under which the minimum of ρ, (cid:126)θ − s (cid:55)→
L OSS s
[Equation]
cannot be reached in D .
This is equivalent to what the theorem we needed to prove states.
B. P ROOF OF REDUCTION FROM MODEL ATTACK TO GRADIENT ATTACK
In this section, we prove a slightly more general result than Lemma 3.
Namely, instead of working with speciﬁc regularizations, we consider a more general class of regularizations, identiﬁed by
Assumption 1.
Lemma 15 (Reduction from model attack to gradient attack) .
Under Assumption 1, if g ts converges and if η t = η is a constant small enough, then ρ t will converge too.
Denote ρ ∞ its limit.
Then for any ε > 0 , there is θ ♠ s ∈ R d such that
[Equation]
Note that since (cid:96) 22 and smooth- (cid:96) 2 regularizations satisfy Assumption 1, Lemma 15 clearly implies
Lemma 3.
We now introduce the key objects of the proof of Lemma 15.
Denote g ∞ s the limit of the attack gradients g ts .
We now deﬁne
[Equation]
and prove that ρ t will converge to the minimizer of L OSS 1 s ( ρ ) .
By Lemma 13, we show that
L OSS 1 s ( ρ ) is both locally strongly convex and L -smooth.
Now deﬁne ζ ts (cid:44) g ts − g ∞ s .
We then know ζ ts → 0 and ∇ L OSS 1 s ( ρ t ) is the sum of all gradient vectors received from all users assuming the strategic user s sends the vector g ∞ s in all iterations.
Thus, at iteration t of the optimization algorithm, we will take one step in the direction G t (cid:44) ∇ L OSS 1 s ( ρ t ) + ζ ts , i.e.,
[Equation] [Equation]
We now prove the following lemma that bounds the difference between the function value in two successive iterations.
Lemma 16.
If L OSS 1 s ( ρ ) is L -smooth and η t ≤ 1 /L , we have
[Equation] [Equation]
Proof.
Since L OSS 1 s is L -smooth, we have
[Equation]
[Equation]
Now plugging ρ t +1 − ρ t = − η t G t and ∇ L OSS 1 s ( ρ t ) =
G t − ζ t s into the inequality implies
[Equation]
where we used the fact η t ≤ 1 /L .
B.3. T HE GLOBAL MODEL REMAINS BOUNDED
Lemma 17.
There is M such that, for all t , L OSS 1 s ( ρ t ) ≤ M .
Proof.
Consider the closed ball B ( ρ ∗ , 1) centered on ρ ∗ and of radius 1.
By Lemma 13, we know that L OSS 1 s is locally strongly convex and thus there exists a µ 1 > 0 such that L OSS 1 s is µ 1 -strongly convex on B ( ρ ∗ , 1) .
Now consider a point ρ 1 on the boundary of B ( ρ ∗ , 1) .
By strong convexity we have
[Equation]
Now similarly, by the convexity of L OSS 1 s on R d , for any ρ ∈ R d − B ( ρ ∗ , 1) , we have (cid:13)(cid:13) ∇ L OSS 1 s ( ρ 1 ) (cid:13)(cid:13) 2 ≥ √ µ 1 .
Now since ζ ts → 0 , there exists an iteration T 1 after which ( t ≥ T 1 ), we have (cid:107) ζ ts (cid:107) 2 ≤ 14 √ µ 1 , and thus (cid:107) G t (cid:107) 2 ≥ (cid:13)(cid:13) ∇ L OSS 1 s ( ρ t ) (cid:13)(cid:13) 2 − (cid:107) ζ ts (cid:107) 2 ≥ 34 √ µ 1 .
Thus, Lemmaimplies that for t ≥ T 1 , if (cid:107) ρ t − ρ ∗ (cid:107) 2 ≥ 1 , then
[Equation] [Equation]
Thus, for (cid:107) ρ t − ρ ∗ (cid:107) 2 ≥ 1 , the loss cannot increase at the next iteration.
Now consider the case (cid:107) ρ t − ρ ∗ (cid:107) 2 < 1 for t ≥ T 1 .
The smoothness of L OSS 1 s implies (cid:13)(cid:13) ∇ L OSS
[Equation]
s ( ρ t ) (cid:13)(cid:13) 2 < L .
Therefore,
[Equation] [Equation]
Now we deﬁne M 1 (cid:44) max ρ ∈B ( ρ ∗ , 1+ η ( L + 14 √ µ 1 ) )
L OSS 1 s ( ρ ) , the maximum function value in the closed ball B (cid:0) ρ ∗ , 1 + η ( L + 14 √ µ 1 ) (cid:1) .
Therefore, we have L OSS 1 s ( ρ t +1 ) ≤ M 1 .
So far we proved that for t ≥ T 1 , in each iteration of gradient descent either the function value will not increase or it will be upper-bounded by M 1 .
This implies that for all t , the function value L OSS 1 s ( ρ t ) is upper-bounded by
[Equation]
This concludes the proof.
Lemma 18.
There is a compact set X such that, for all t , ρ t ∈ X .
Proof.
Now since L OSS 1 s is µ 1 -strongly convex in B ( ρ ∗ , 1) , for any point ρ ∈ R d such that
[Equation]
ρ − ρ t (cid:107) 2 = 1 , we have
[Equation]
But now by the convexity of L OSS 1 s in R d , for any ρ such that (cid:107) ρ − ρ ∗ (cid:107) 2 ≥ 1 , we have
[Equation]
This implies that if (cid:107) ρ t − ρ ∗ (cid:107) 2 > 2 µ 1 (cid:0) M 2 − L OSS 1 s ( ρ ∗ ) (cid:1) , then L OSS 1 s ( ρ t ) > M 2 .
Therefore, we must have (cid:107) ρ t − ρ ∗ (cid:107) 2 ≤ 2 µ 1 (cid:0) M 2 − L OSS 1 s ( ρ ∗ ) (cid:1) , for all t ≥ 0 .
This describes a closed ball, which is a compact set.
B.3. C ONVERGENCE OF THE GLOBAL MODEL UNDER CONVERGING GRADIENT ATTACK
Lemma 19.
Suppose u t ≥ 0 veriﬁes u t +1 ≤ αu t + δ t , with δ t → 0 .
Then u t → 0 .
Proof.
We now show that for any ε > 0 , there exists an iteration T ( ε ) , such that for t ≥ T ( ε ) , we have u t ≤ ε .
For this, note that by induction, we observe that, for all t ≥ 0 ,
[Equation]
Since δ t → 0 , there exists an iteration T 2 ( ε ) such that for all t ≥ T 2 ( ε ) , we have δ t ≤ ε (1 − α ).
Therefore, for t ≥ T 2 ( ε ) , we have
[Equation] [Equation]
[Equation]
Denoting M 0 ( ε ) (cid:44) (cid:80) T 2 (
[Equation]
−
[Equation]
= α
[Equation]
s
[Equation]
1 δ s , we then have
[Equation]
Therefore, for t ≥ ln
[Equation]
, we have
[Equation] [Equation]
This proves that u t → 0 .
We ﬁrst prove the ﬁrst part of Lemma 3.
Lemma 20.
Under Assumption 1 and η t = η ≤ 1 /L , if g ts converges, then so does ρ t .
Proof.
Deﬁne X based on Lemma 18.
Since L OSS 1 s is locally strongly convex, there exists µ 2 >such that L OSS 1 s is µ 2 -strongly convex in a convex compact set X containing ρ t for all t ≥ 0 .
By the strong convexity of L OSS 1 s ( ρ ) , we have
[Equation] [Equation]
Now, using the fact
[Equation]
we have
[Equation] [Equation]
[Equation]
But now note that L OSS 1 s ( ρ t ) − L OSS 1 s ( ρ ∗ ) ≥ L OSS 1 s ( ρ t ) − L OSS 1 s ( ρ t +1 ) .
Thus, combining Equation (77) and Lemma 16 yields
[Equation]
By rearranging the terms
[Equation]
we then have
[Equation]
Now note that η ≤ 1 /L < 1 /µ 2 and thus 0 < 1 − µ 2 η < 1 .
We now deﬁne two sequences u t (cid:44) (cid:107) ρ t − ρ ∗ (cid:107) 2 and δ t = η (cid:107) ζ ts (cid:107) 2 .
We already know that δ t → 0 , and we want to show u t also converges to 0 .
By Equation (80), we have
[Equation]
81) which implies
[Equation]
and thus u
[Equation]
Lemma 19 allows to conclude.
B.3. R EDUCTION FROM MODEL ATTACK TO CONVERGING GRADIENT ATTACK
Proof of Lemma 15.
Lemma 20 already provided the convergence part of Lemma 3.
We now move forward to proving the second part of the theorem, showing that for any ε > 0 , there exists θ ♠ s ∈ R d such that (cid:13)(cid:13)(cid:13) ρ ∞ − ρ ∗ ( θ ♠ s , (cid:126) D − s ) (cid:13)(cid:13)(cid:13) 2 ≤ ε .
We deﬁne
[Equation]
and ρ ∗ ( θ s ) , its minimizer.
Therefore, we have
[Equation]
By Lemma 13, we know that L OSS 2 s is locally strongly convex.
Therefore, there exists µ 3 > 0 such that L OSS 2 s ( ρ ) is µ 3 -strongly convex in { ρ | (cid:107) ρ − ρ ∗ ( θ s ) (cid:107) 2 ≤ 1 } .
Therefore, for any 0 < ε < 1 , if
[Equation]
ρ ∞ − ρ ∗ ( θ s ) (cid:107) 2 > ε , we then have
[Equation] [Equation]
[Equation]
and thus (cid:13)(cid:13) ∇ ρ L OSS 2 s ( ρ ∞ , θ ♠ s ) (cid:13)(cid:13) 2 ≥ µ 3 ε .
Now since g ∞ s ∈ G RAD ( ρ ∞ ) there exists θ ♠ s ∈ R d such that 3 (cid:13)(cid:13) ∇ ρ R ( ρ ∞ , θ ♠ s ) − g ∞ s (cid:13)(cid:13) 2 ≤ µ 3 εwhich yields
[Equation]
which is a contradiction.
Therefore, we must have (cid:13)(cid:13)(cid:13) ρ ∞ − ρ ∗ ( θ ♠ s ,
[Equation]
D − s ) (cid:13)(cid:13)(cid:13) 2 ≤ ε .
C S UM OVER EXPECTATIONS
In this section, we provide both theoretical and empirical results to argue for using a sum-based local loss over an expectation-based local loss.
C. T HEORETICAL ARGUMENTS
Indeed, intuitively, if one considers an expectation E x ∼D n [ (cid:96) ( θ n , x )] rather than a sum, as is done by
Hanzely et al.
Dinh et al.
and El-Mhamdi et al.
(2021), then the weight of an honest active user’s local loss will not increase as a node provides more and more data, which will hinder the ability of θ n to ﬁt the user’s local data.
In fact, intuitively, using an expectation wrongly yields the same inﬂuence to any two nodes, even when one (honest) node provides a much larger dataset
D n than the other, and should thus intuitively be regarded as “more reliable”.
There is another theoretical argument for using the sum rather than the expectation.
Namely, if the loss is regarded as a Bayesian negative log-posterior, given a prior exp (cid:16) − (cid:80) n ∈ [ N ] ν (cid:107) θ n (cid:107) 2 − (cid:80) n ∈
[ N ] R ( ρ, θ n ) (cid:17) on the local and global models, then the term that ﬁts local data should equal the negative log-likelihood of the data, given the models ( ρ, (cid:126) θ ) .
Assuming that the distribution of each data point x ∈ D n is independent from all other data points, and depends only on the local model θ n , this negative log-likelihood yields a sum over data points; not an expectation.
C. E MPIRICAL RESULTS
We also empirically compared the performances of sum as opposed to the expectation.
To do so, we constructed a setting where 10 “idle” users draw randomly 10 data points from the FashionMNIST dataset, while one “active” user has all of the FashionMNIST dataset (60,000 data points).
We then learned local and global models, with R ( ρ, θ ) (cid:44)
λ (cid:107) ρ − θ (cid:107) 22 , λ
=
We compared two different classiﬁers to which we refer as a “linear model” and “2-layers neural network”, both using
CrossEntropy loss.
The linear model has (784+1) × 10 parameters.
The neural network has 2 layers of 784 parameters with bias, with ReLU activation in between, adding up to ( (784 + 1) × 784 + (784 + 1) × 10 .
Note also that, in all our experiments, we did not consider any local regularization, i.e. we set ν (cid:44) 0 .
All our experiments are seeded with seed 999.
C.2. N OISY F ASHION MNIST
To see a strong difference between sum and average, we made the FashionMNIST dataset harder to learn, by randomly labeling 60% of the training set.
Table 1 reports the accuracy of local and global models in the different settings.
Our results clearly and robustly indicate that the use of sums outperforms the use of expectations.
[Table]
and
(in Table 1, the mean is reported), as we vary the value of λ .
Intuitively, λ models how much we want the local models to be similar
In the case of learning FashionMNIST, given that the data is i
[Figure]
i
[Figure]
d., larger values of λ are more meaningful (though our experiments show that they may hinder convergence speed).
However, in settings where users have different data distributions, e
[Figure]
g. because the labels depend on users’ preferences, then smaller values of λ may be more relevant.
Note that the use of a common value of λ in both cases is slightly misleading, as using the sum intuitively decreases the comparative weight of the regularization term.
To reduce this effect, for this experiment only, we divide the local losses by the average of the number of data points per node for the sum version.
This way, if the number of points is equal for all nodes, the two losses will be exactly the same.
What’s more, our experiments seem to robustly show that using the sum consistently outperforms the expectation, for both a linear classiﬁer and a 2-layer neural network, for the problem of noisy FashionMNIST classiﬁcation
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
C.2. F ASHION MNIST WITHOUT NOISE
Recall that we introduced noise into FashionMNIST to make the problem harder to learn and observe a clear difference between the average and the sum.
In this section
[Figure]
we present results of our experiments when the noise is removed
[Figure] [Figure]
Even without noise, the difference between using the sum and using the expectation still seems important.
We acknowledge, however, that the plots suggest that even though we ran this experiment for 10 times more (and 5 times more for the linear model) than other experiments, we might not have reached convergence yet, and that the use of the expectation might still eventually gets closer to the case of sum.
We believe that the fact that the difference between sum and expectation in the absence of noise is weak is due to the fact that the FashionMNIST dataset is sufﬁciently linearly separable.
Thus, we achieve a near-zero loss in both cases, which make the sum and the expectation close at optimum.
Even in this case, however, we observed that the sum clearly outperforms the expectation especially, in the ﬁrst epochs.
We argue that the reason for this is the following.
By taking the average in local losses, the weights of the data of idle nodes are essentially blown out of proportion.
As a result, the optimizer will very quickly ﬁt these data.
However, the signal from the data of the active node will then be too weak, so that the optimizer has to ﬁrst almost perfectly ﬁt the idle nodes’ data before it can catch the signal of the active node’s data and hence the average achieves weaker convergence performances than the sum.
D L INEAR REGRESSION AND CLASSIFICATION ARE GRADIENT PAC*
Throughout this section, we use the following terminology.
Deﬁnition 5.
Consider a parameterized event E ( I ) .
We say that the event E occurs with high probability if P [ E ( I )]
→ 1 as I → ∞ .
D. P RELIMINARIES
Deﬁne
[Equation]
max (cid:107)
[Equation] [Equation]
=
[Equation]
( (cid:107) Σ x (cid:107) 2 / (cid:107) x (cid:107) 2 ) the (cid:96) 2 operator norm of the matrix Σ .
For symmetric matrices Σ , this is also the largest eigenvalue in absolute value.
Theorem 6 (Covariance concentration, Theorem 6.5 in Wainwright (2019)) .
Denote Σ =
E (cid:2) Q i Q Ti (cid:3) , where
Q i ∈ R d is from a σ Q -sub-Gaussian random distribution ˜ Q .
Then, there are universal constants c 1 , c 2 and c 3 such that, for any set {Q i } i ∈
[ I ] of i.i.d.
samples from ˜ Q , and any δ > 0 , the sample covariance (cid:98) Σ
= 1 I (cid:80) Q i Q Ti satisﬁes the bound
[Equation]
Theorem 7 (Weyl’s Theorem, Theorem 4.3.1 in Horn & Johnson (2012)) .
Let A and B be Hermitian 4 and let the respective eigenvalues of A and B and A + B be { λ i ( A ) } di =1 , { λ i ( B ) } di =1 , and
For
{ λ i ( A + B ) } di =1 , each increasingly ordered.
Then
[Equation]
and
[Equation]
for each i = 1 , ..., d .
Lemma 21.
Consider two symmetric deﬁnite positive matrices S and Σ .
Denote ρ min and λ min their minimal eigenvalues.
Then | ρ min − λ min | ≤ (cid:107) S − Σ (cid:107) 2 .
Proof.
This is a direct consequence of Theorem 7, for A = S , B = Σ − S , i = 1 , and j = 0 .
Corollary 1.
There are universal constants c 1 , c 2 and c 3 such that, for any σ Q -sub-Gaussian vector distribution ˜ Q ∈ R d and any δ > 0 , the sample covariance (cid:98) Σ = 1 I (cid:80) Q i Q Ti satisﬁes the bound
P (cid:34)
[Equation]
where min S P ( ˆ Σ) and min S P (Σ) are the minimal eigenvalues of ˆ Σ and Σ .
Proof.
This follows from Theorem 6 and Lemma 21.
Lemma 22.
With high probability, min S P (ˆΣ) ≥ min S P (Σ) / 2 .
Proof.
Denote λ min (cid:44) min S P (Σ) and (cid:98) λ min (cid:44) min S P (ˆΣ) .
Since each Q i is drawn i.i.d.
from a σ Q -sub-Gaussian, we can apply Corollary 1.
Namely, there are constants c 1 , c 2 and c 3 , such that for any δ > 0 , we have
[Equation]
We now set δ (cid:44) λ min / (4 σ 2 Q ) and we consider I large enough so that c 1 (cid:16)(cid:113)
d I + d I (cid:17) ≤ λ min / (4 σ 2 Q ) .
With high probability, we then have (cid:98) λ min ≥ λ min / 2 .
D. L INEAR REGRESSION IS GRADIENT -PAC*
In this section, we prove the ﬁrst part of Lemma 4.
Namely, we prove that linear regression is gradient-PAC* learning.
D.2. L EMMAS FOR LINEAR REGRESSION
Before moving to the main proof that linear regression is gradient-PAC*, we ﬁrst prove a few useful lemmas.
These lemmas will rest on the following well-known theorems.
Theorem 8 (Lemma 2.7.7 in Vershynin (2018)) .
If X and Y are sub-Gaussian, then XY is subexponential.
Theorem 9 (Equation 2.18 in Wainwright (2019)) .
If X 1 , . .
, X I are iid sub-exponential variables, then there exist constants c 4 , c 5 such that, for all I , we have
[Equation] [Equation]
Lemma 23.
For all j ∈ [ d ] , the random variables X i (cid:44) ξ i Q i [ j ] are iid, sub-exponential and have zero mean.
Proof.
The fact that these variables are iid follows straightforwardly from the fact that the noises ξ i are iid, and the queries Q i are also iid.
Moreover, both are sub-Gaussian, and by Theorem 8, the product of sub-Gaussian variables is sub-exponential.
Finally, we have E [ X ] =
E [ ξ Q [ j ]] =
E [ ξ ] E [ Q [ j ]] = 0 , using the independence of the noise and the query, and the fact that noises have zero mean ( E [ ξ ] = 0 ).
Lemma 24.
There exists B such that (cid:13)(cid:13)(cid:80) i ∈I
ξ i Q i (cid:13)(cid:13) 2 ≤ B I 3 / 4 with high probability.
Proof.
By Lemma 23, the terms ξ i Q i
[ j ] are iid, sub-exponential and have zero mean.
Therefore, by
Theorem 9, there exist constants c 4 and c 5 such that for any coordinate j ∈ [ d ] of ξ i Q i and for all 0 ≤ u ≤ c 4 , we have
[Equation]
Plugging u
= v I ( − 1 / 4) into the inequality for some small enough constant v , and using union bound then yields
[Equation]
Deﬁning B (cid:44) v √ d yields the lemma.
D.2. P ROOF THAT LINEAR REGRESSION IS GRADIENT -PAC*
We now move on to proving that least square linear regression is gradient-PAC*.
Proof of Theorem 2.
Note that ∇ θ (cid:96) ( θ, Q , A ) =
( θ T Q − A ) Q .
Thus, on input i ∈
[ I ] , we have
[Equation]
Moreover, we have
[Equation]
As a result, we have
[Equation]
But now, with high probability, we have ( θ − θ † ) T (cid:98) Σ( θ − θ † ) ≥ ( λ
[Equation] [Equation]
θ − θ † (cid:13)(cid:13) 2 2 (Lemma 22) and (cid:13)(cid:13)(cid:80) i ∈I ξ
i Q i (cid:13)(cid:13) 2 ≤ B I (3 / 4) (Lemma 24).
Using the fact that (cid:13)(cid:13) θ † (cid:13)(cid:13) 2 ≤ K and the Cauchy-Schwarz inequality, we have
[Equation]
Denoting A K (cid:44) λ minand B K (cid:44) B + 2 ν K and using the fact that I ≥ 1 , we then have
[Equation]
with high probability.
This corresponds to saying Assumption 2 is satisﬁed for α = 3 / 4 .
D. L OGISTIC REGRESSION
In this section, we now prove the second part of Lemma 4.
Namely, we prove that logistic regression is gradient-PAC* learning.
D.3. L EMMAS ABOUT THE SIGMOID FUNCTION
We ﬁrst prove two useful lemmas about the following logistic distance function.
Deﬁnition 6.
We deﬁne the logistic distance function by ∆( a, b ) (cid:44) ( a − b ) ( σ ( a ) − σ ( b )) .
Lemma 25.
If a, b ∈ R such that for some k > 0 , | a | ≤ k and | b | ≤ k , then there exists some constant c k > 0 such that
[Equation]
Proof.
Note that the derivative of σ ( z ) is strictly positive, symmetric ( σ (cid:48)
( z ) =
σ (cid:48) ( − z ) ) and monotonically decreasing for z ≥ 0 .
Therefore, for any z ∈ [ − k, k ] , we know σ (cid:48)
( z ) ≥ c k (cid:44) σ (cid:48) ( k ) .
Thus, by the mean value theorem, we have
[Equation]
Multiplying both sides by ( a − b ) 2 then yields the lemma.
Lemma 26.
If b ∈ R , and | b | ≤ k , for some k > 0 , then there exists a constant d k , such that for any a ∈ R , we have
[Equation]
Proof.
Assume | a − b | ≥ 1 and deﬁne d k (cid:44) σ ( k + 1) − σ ( k ) .
If b ≥ 0 , since σ (cid:48) ( z ) is decreasing for z ≥ 0 , we have σ ( b ) − σ ( b − 1) ≥ σ ( b + 1) − σ ( b ) ≥ d k , and by symmetry, a similar argument holds for b ≤ 0 .
Thus, we have
[Equation]
Therefore,
[Equation] [Equation]
For the case of | a − b | ≤ 1 , we also have ( a − b ) ( σ ( a ) − σ ( b )) ≥ 0 ≥ d k | a − b | − d k .
D.3.
A UNIFORM LOWER BOUND
Deﬁnition 7.
Denote S d − 1 (cid:44) (cid:8) u ∈ R d (cid:12)(cid:12) (cid:107) u (cid:107) 2 = 1 (cid:9) the hypersphere in R d .
Lemma 27.
Assume S UPP ( ˜ Q ) spans R d .
Then, for all u ∈ S d − 1 , E (cid:2)(cid:12)(cid:12) Q T u (cid:12)(cid:12)(cid:3) > 0 .
Proof.
Let u ∈ S
d − 1 .
We know that there exists Q 1 , . . .
, Q d ∈ S UPP ( ˜ Q ) and α 1 , . . . , α d ∈ R such that u is colinear with (cid:80)
α j Q j .
In particular, we then have u T (cid:80) α j Q j = (cid:80) α j ( Q Tj u ) (cid:54) = 0 .
Therefore, there must be a query Q ∗ ∈ S UPP ( ˜ Q ) such that Q T ∗ u (cid:54) = 0 , which implies a (cid:44) (cid:12) (cid:12) Q T ∗ u (cid:12) (cid:12) >
By continuity of the scalar product, there must then also exist ε > 0 such that, for any Q ∈
B ( Q ∗ , ε ) , we have (cid:12)(cid:12) Q T u (cid:12)(cid:12) ≥ a/ 2 , where B ( Q ∗ , ε ) is an Euclidean ball centered on Q ∗ and of radius ε .
But now, by deﬁnition of the support, we know that p (cid:44) P [ Q ∈ B ( Q ∗ , ε )] > 0 .
By the law of total expectation, we then have
[Equation]
which is the lemma.
Lemma 28.
Assume that, for all unit vectors u ∈ S d − 1 , we have E (cid:2)(cid:12)(cid:12) Q T u (cid:12)(cid:12)(cid:3) > 0 , and that S UPP ( ˜ Q ) is bounded by M Q .
Then there exists C > 0 such that, with high probability,
[Equation]
Proof.
By continuity of the scalar product and the expectation operator, and by compactness of S d − 1 , we know that
[Equation]
Now deﬁne ε (cid:44) C 0 / 4 M Q .
Note that S d − 1 ⊂ (cid:83) u ∈ S
d − 1 B ( u , ε ) .
Thus we have a covering of the hypersphere by open sets.
But since S d − 1 is compact, we know that we can extract a ﬁnite covering.
In other words, there exists a ﬁnite subset S ⊂ S d − 1 such that S d − 1 ⊂ (cid:83) u ∈ S B ( u , ε ) .
Put differently, for any v ∈ S
d − 1 , there exists u ∈ S such that (cid:107) u − v (cid:107) 2 ≤ ε .
Now consider u ∈ S .
Given that S UPP ( ˜ Q ) is bounded, we know that (cid:12)(cid:12) Q Ti u (cid:12)(cid:12) ∈
[0 , M Q ] .
Moreover, such variables (cid:12)(cid:12) Q Ti u (cid:12)(cid:12) are iid.
By Hoeffding’s inequality, for any t > 0 , we have
[Equation]
Choosing t =
C 0 / 2 then yields
[Equation]
Taking a union bound for u ∈ S then guarantees
[Equation]
which clearly goes to 1 as I → ∞ .
Thus ∀ u ∈ S, (cid:80) i ∈I (cid:12)(cid:12) Q Ti u (cid:12)(cid:12) ≥ C
I 2 holds with high probability.
Now consider v ∈ S d − 1 .
We know that there exists u ∈ S such that (cid:107) u − v (cid:107) 2 ≤ ε .
Then, we have (cid:88)
[Equation] [Equation]
which proves the lemma.
D.3. L OWER BOUND ON THE DISCREPANCY BETWEEN PREFERRED AND REPORTED ANSWERS
Lemma 29.
Assume that ˜ Q has a bounded support, whose interior contains the origin.
Suppose also that (cid:13)(cid:13) θ † (cid:13)(cid:13) 2 ≤ K .
Then there exists A K such that, with high probability, we have
[Equation]
Proof.
Note that by Cauchy-Schwarz inequality we have
[Equation]
Thus
[Equation]
Lemma 26 implies the existence of a positive constant d K , such that for all θ ∈ R d , we have (cid:88)
[Equation]
[Equation]
where u θ − θ † (cid:44) ( θ − θ † ) /
(cid:13)(cid:13)
θ − θ † (cid:13)(cid:13) 2 is the unit vector in the direction of θ − θ † .
Now, by Lemma 28, we know that, with high probability, for all unit vectors u ∈ S
d − 1 , we have (cid:80) (cid:12)(cid:12)
Q Ti u (cid:12)(cid:12) ≥ C I .
Thus, for I sufﬁciently large, for any θ ∈ R d , with high probability, we have (cid:88) i ∈I
[Equation]
I (cid:13)(cid:13)
[Equation]
(cid:13)(cid:13) 2 − d K I .
Deﬁning e K (cid:44) d K C min, and f K (cid:44) 4 C min , for (cid:13)(cid:13)
θ − θ † (cid:13)(cid:13) 2 > f K , we then have (cid:88) i ∈I ∆( Q
[Equation] [Equation]
−
[Equation]
(cid:13)(cid:13) 2 .
We now focus on the case of (cid:13)(cid:13)
θ − θ † (cid:13)(cid:13) 2 ≤ f K .
The triangle inequality yields (cid:107) θ (cid:107) 2 ≤ (cid:13)(cid:13)
θ − θ † (cid:13)(cid:13) 2 + (cid:13)(cid:13) θ † (cid:13)(cid:13) 2 ≤ f K + K .
By Cauchy-Schwarz inequality, we then have (cid:12)(cid:12) Q Ti θ (cid:12)(cid:12) ≤ ( f K + K ) M Q (cid:44) g K and (cid:12) (cid:12) Q T i θ † (cid:12) (cid:12) ≤ K M Q ≤ g K .
Thus, by Lemma 25, we know there exists some constant c K such that
[Equation]
Since distribution ˜ Q is bounded (and thus sub-Gaussian), by Theorem 6, with high probability, we have
[Equation]
where λ min is the smallest eigenvalue of E (cid:2) Q i Q Ti (cid:3) .
Thus, for (cid:13)(cid:13) θ − θ † (cid:13)(cid:13) 2 ≤ f K , we have (cid:88)
[Equation]
∈I
[Equation]
Combining this with (127), and deﬁning A K (cid:44) min (cid:8) λ min c K, e K (cid:9) , we then obtain the lemma.
D.3. P ROOF THAT LOGISTIC REGRESSION IS GRADIENT -PAC*
Now we proceed with the proof that logistic regression is gradient-PAC*.
Proof of Theorem 3.
Note that σ ( − z ) =
e − z σ ( z ) = 1 − σ ( z ) and σ (cid:48)
( z ) =
e − z σ 2 ( z ) .
We then have
[Equation]
where 1 [ A = 1] is the indicator function that outputs 1 if A = 1 , and 0 otherwise.
As a result,
[Equation]
By Lemma
[Equation]
with high probability, we have
[Equation]
[ I ]
[Equation]
To control the second term of (139), note that the random vectors Z i (cid:44) (cid:0) σ ( Q Ti θ † ) −
[ A i = 1] (cid:1) Q i are iid with norm at most M Q .
Moreover, since E [ 1 [ A i = 1] |Q i ]
= σ ( Q Ti θ † ) , by the tower rule, we have E [ Z i ] =
E [ E [ Z i |Q i ]] = 0 .
Therefore, by applying Hoeffding’s bound to every coordinate of Z i , and then taking a union bound, for any B > 0 , we have
[Equation]
Applying now Cauchy-Schwarz inequality, with high probability, we have
[Equation]
Combining this with (132) and using (cid:13)(cid:13) θ † (cid:13)(cid:13) 2 2 ≤ K , we then have
[Equation]
[Equation] [Equation]
(145) where B K = B + 2 ν K .
This shows that Assumption 2 is satisﬁed for logistic loss for α = 3 / 4 , and
A K and B K as previously deﬁned.
E P ROOFS OF LOCAL PAC*- LEARNABILITY
Let us now prove Lemma 5.
To do so, consider the preferred models (cid:126)θ † and a subset H ⊂
[ N ] of honest users.
Denote (cid:126) D − H the datasets provided by users n ∈ [ N ] − H .
Each honest user h ∈ H provides an honest dataset D h of cardinality at least I ≥ 1 .
Consider the bound K H (cid:44) max h ∈ H (cid:13)(cid:13)(cid:13) θ † h (cid:13)(cid:13)(cid:13) 2 on the parameter norm of honest active users h ∈ H .
E. B OUNDS ON THE OPTIMA
Before proving the theorem, we prove a useful lemma that bounds the set of possible values for the global model and honest local models.
Lemma 30.
Assume that R and (cid:96) are nonnegative.
For I large enough, if all honest active nodes h ∈ H provide at least I data, then, with high probability, (cid:126)θ ∗ H must lie in a compact subset of R d × H that does not depend on I .
Proof.
Denote L 0 (cid:44) L OSS (0 , ( (cid:126)θ † H , 0 − H ) , ( ∅ , (cid:126) D − H )) .
Essentially, we will show that, if (cid:126)θ ∗ H is too far from (cid:126)θ † H , then the loss will take values strictly larger than L 0 .
Assumption 2 implies the existence of an event E that occurs with probability at least P 0 (cid:44)
P ( K H , I ) | H | , under which, for any θ h ∈ R d , we have
[Equation]
which implies
[Equation] [Equation]
Note also that P 0 → 1 as I → ∞ .
We now integrate both sides over the line segment from θ † h to θ h .
The fundamental theorem of calculus for line integrals
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]
[Equation]
Now for I > I 1 (cid:44) max (cid:110) 2 L 0 /A
K H , (4 B K H /A
K H )
1 1 − α (cid:111) , we have
[Equation]
This implies that if (cid:13)(cid:13)(cid:13) θ h − θ † h (cid:13)(cid:13)(cid:13) 2 > 2 for any h ∈ H , then we have
[Equation] [Equation]
regardless of ρ and θ − H .
Therefore, we must have (cid:13)(cid:13)(cid:13) θ † h − θ ∗ h (cid:13)(cid:13)(cid:13) 2 ≤ 2 .
Such inequalities describe a bounded closed subset of R d × H , which is thus compact.
Lemma 31.
Assume that R ( ρ, θ ) → ∞ as (cid:107) ρ − θ (cid:107) 2 → ∞ , and that (cid:13)(cid:13)(cid:13) θ † h − θ ∗ h (cid:13)(cid:13)(cid:13) 2 ≤ 2 for all honest users h ∈ H .
Then ρ ∗ must lie in a compact subset of R d that does not depend on I .
Proof.
Consider an honest user h (cid:48) .
Given our assumption on R → ∞ , we know that there exists
D K H such that if (cid:107) ρ − θ ∗ h (cid:48) (cid:107) 2 ≥ D K H , then R ( ρ, θ ∗ h (cid:48) ) ≥ L 0 + 1 .
Thus any global optimum ρ ∗ must
[Table]
E. P ROOF OF L EMMA
Proof of Lemma 5.
Fix ε, δ > 0 .
We want to show the existence of some value of I ( ε, δ, (cid:126) D − H , (cid:126)θ † ) that will guarantee ( ε, δ ) -locally
PAC* learning for honest users.
By lemmas 30 and 31, we know that the set C of possible values for ( ρ ∗ , (cid:126)θ ∗ H ) is compact.
Now, we deﬁne
[Equation]
155) the maximum of the norm of achievable gradients at the optimum.
We know this maximum exists since C is compact.
Using the optimality of ( ρ ∗ , (cid:126)θ ∗ ) , for all h ∈ H , we have
[Equation] [Equation]
[Equation] [Equation]
We now apply assumption 2 for θ = θ ∗ h (for h ∈ H ).
Thus, there exists some other event E (cid:48) with probability at least P 0 , under which, for all h ∈ H , we have 0 ≥ A K
[Equation]
I min (cid:26)(cid:13)(cid:13)(cid:13) θ ∗ h − θ † h (cid:13)(cid:13)(cid:13) 2 , (cid:13)(cid:13)(cid:13) θ ∗ h − θ † h (cid:13)(cid:13)(cid:13) 2(cid:27)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
h (cid:13)(cid:13)(cid:13)
[Equation]
− E
[Equation]
(cid:13)(cid:13)(cid:13) θ
[Equation]
θ
[Equation]
h (cid:13)(cid:13)(cid:13) 2 .
Now if I > I 2 (cid:44) max (cid:110) 2 E K H /A
K H , (2 B K H /A
K H )
1 1 − α (cid:111)
this inequality cannot hold for (cid:13)(cid:13)(cid:13) θ ∗ h − θ † h (cid:13)(cid:13)(cid:13) 2 ≥ 1 .
Therefore, for I > I 2 , we have (cid:13)(cid:13)(cid:13) θ ∗ h − θ † h (cid:13)(cid:13)(cid:13) 2 < 1 , and thus,
[Equation]
(cid:13)(cid:13)(cid:13) θ ∗ h − θ † h (cid:13)(cid:13)(cid:13) 2 2 − B K H I α (cid:13)(cid:13)(cid:13) θ ∗ h − θ † h (cid:13)(cid:13)(cid:13) 2 − E K H (cid:13)(cid:13)(cid:13) θ ∗ h − θ † h (cid:13)(cid:13)(cid:13)(161) and thus, (cid:13)(cid:13)(cid:13) θ ∗ h − θ † h (cid:13)(cid:13)(cid:13)
[Equation]
Now note that P [ E ∧ E (cid:48) ]
= 1 − P [ ¬E ∨ ¬E (cid:48) ] ≥ 1 − P [ ¬E ] − P [ ¬E (cid:48) ]
= 2 P 0 − 1 .
It now sufﬁces to consider I larger than I 2 and large enough so that P ( K H , I ) | H | ≥ 1 − δ/ 2 (whose existence is guaranteed by Assumption 2, and which guarantees 2 P 0 − 1 ≥ 1 − δ ) and so that B KH I α + E KH
A KH I ≤ ε to obtain the theorem.
F C ONVERGENCE OF CGA AGAINST (cid:96)
To write our proof, we deﬁne L OSS ρ − s : R d → R by
[Equation]
In other words, it is the loss when local models are optimized, and when the data of strategic user s are removed.
Lemma 32.
Assuming (cid:96) 2 2 regularization and convex loss-per-input functions (cid:96) , for any datasets (cid:126) D ,
L OSS is strongly convex.
As a result, so is L OSS ρ − s .
Proof.
Note that the global loss can be written as a sum of convex function, and of ν (cid:80) (cid:107) θ n (cid:107) 22 + (cid:107) ρ − θ 1 (cid:107) 22 .
Using tricks similar to the proof of Lemma 11, we see that the loss is strongly convex.
The latter part of the lemma is then a straightforward application of Lemma 10.
We now move on to the proof of Theorem 4.
Note that our statement of the proof was not fully explicit, especially about the upper bound on the constant learning rate η .
Here, we prove that it holds for η t = η ≤ 1 / 3 L , where L is a constant such that L OSS ρ − s is L -smooth.
The existence of L is guaranteed by Lemma 13.
Proof of Theorem 4.
Note that by Lemma 9, L OSS ρ − s is convex, differentiable and L -smooth, and ∇ L OSS ρ − s ( ρ t )
= g † ,t − s .
For (cid:96) 22 regularization, we have G RAD ( ρ ) =
R d for all ρ ∈ R d .
Then the minimum of equation 6 is zero, which is obtained when g ts (cid:44) ρ t − θ † s η
[Equation]
Note that
[Equation] [Equation]
[Equation]
Therefore, ρ t +1 − ρ t = η ( g † ,t − s − g † ,t
[Equation] [Equation]
s ) −
[Equation]
Then, using the L -smoothness of L OSS ρ − s , and denoting u t (cid:44) (cid:13)(cid:13) ρ t +1 − ρ t (cid:13)(cid:13) 2 , we have u t +1 ≤
Lη t u t + Lη t − 1 u t − 1 .
Now assume that η ≤ 1 / 3 L .
Then u t +1 ≤ 13 ( u t + u t − 1 ) .
We then know that u t +2 ≤ 13 ( u t +1 + u t ) ≤ 13 ( 13 ( u t + u t − 1 ) + u t ) = 49 u t + 19 u t − 1 .
Now deﬁne v t (cid:44) u t + u t − 1 .
We then have v t +2 ≤ u t +2 + u t +1 ≤ 79 u t + 49 u t − 1 ≤ 79 ( u t + u t − 1 ) ≤ 79 v t .
By induction, we know that v t ≤ (7 / 9) ( t − 1) / 2 max { v 0 , v 1 } ≤ ( √ 7 / 3) t (cid:0) ( √ 7 / 3) max { v 0 , v 1 } (cid:1) .
Thus, deﬁning α (cid:44) √ 7 / 3 < 1 , there exists C > 0 such that u t ≤ v t ≤ Cα t .
This implies that (cid:80) (cid:13)(cid:13) ρ t +1 − ρ t (cid:13)(cid:13) 2 ≤ (cid:80)
Cα t < ∞ .
Thus (cid:80) ( ρ t +1 − ρ t ) converges, which implies the convergence of ρ t to a limit ρ ∞ .
By L -smoothness, we know that g † ,t − s must converge too.
Taking equation 168 to the limit then implies ρ ∞ =
θ † s .
This shows that the strategic user achieves precisely what they want with CGA.
It is thus optimal.
G CGA ON MNIST
In this section, CGA is executed against 10 honest users, each one having 6,000 randomly and data points of MNIST, drawn randomly and independently.
CGA is run by a strategic user whose target model θ † s labels 0’s as 1’s, 1’s as 2’s, and so on, until 9’s as 0’s.
We learn θ † s by relabeling the
MNIST training dataset and learning from the relabeled data.
We use λ = 1 , Adam optimizer and a decreasing learning rate
[Figure]
H S INGLE DATA POISONING FOR LEAST SQUARE LINEAR REGRESSION
Proof of Theorem 5.
We deﬁne the minimized loss with respect to ρ and without strategic user s by
[Equation]
Now consider a subgradient g ∈ ∇ ρ L OSS ∗− s ( θ † s , (cid:126)
D − s ) of the minimized loss at θ † s .
For x (cid:44) − g 2 λ , then have − g ∈ ∇ (cid:16) λ (cid:107) x (cid:107) 22 (cid:17) .
We then deﬁne θ
♠ s (cid:44) θ † s − x .
[Equation]
where L OSS s is deﬁned by (38).
Now consider the data point ( Q , A )
=
( g, g T θ ♠ s − 1) .
For
D s = { ( Q , A ) } , we then have ∇L s ( θ ♠ s , D s ) =
g , which implies
[Equation]
Combining it all together with the uniqueness of the solution then yields arg min
[Equation]
which is what we wanted.
I D ATA POISONING AGAINST LINEAR CLASSIFICATION I. G ENERATING EFFICIENT POISONING DATA
For every label a ∈ { 1 , . . .
, 9 } , we deﬁne y a (cid:44) θ ♠ a − θ ♠ 0 , and c a (cid:44) − ( θ ♠ a 0 − θ ♠ 00 ) (where θ ♠ a 0 is the bias of the linear classiﬁer).
The indifference subspace V is then the set of images Q ∈ R d such that Q T y a =
c a for all a ∈ { 1 , . . .
To project any image X ∈ R d on V , let us ﬁrst construct an orthogonal basis of the vector space orthogonal to V , using the Gram-Schmidt algorithm.
Namely, we ﬁrst deﬁne z 1 (cid:44) y 1 .
Then, for any answer a ∈ { 1 , . .
, 9 } , we deﬁne
[Equation]
It is easy to check that for b < a , we have z T a z b = 0 .
Moreover, if Q ∈ V , then
[Equation]
b
[Equation]
(cid:107)
[Equation]
By induction, we see that z Ta Q is a constant independent from Q .
Indeed, for a = 1 , this is clear as z T 1 Q = y T 1 Q =
c 1 .
Moreover, for a > 1 , then, in the computation of z Ta Q , Q always appear as z Tb Q for b < a .
Moreover, denoting c (cid:48) a the constant such that z Ta Q = c (cid:48) a for all a ∈ { 1 , . . .
9 } , we see that these constants can be computed by
[Equation]
Finally, we can simply perform repeated projection onto the hyperplanes where a is equally probable as the answer 0 .
To do this, we ﬁrst deﬁne the orthogonal projection P ( X, y, c ) of X ∈ R d on the hyperplane x T y =
c , which is given by
[Equation]
It is straightforward to verify that P ( X, y, c ) T y =
c and that P ( P ( X, y, c ) , y, c ) =
P ( X, y, c ) .
We then canonically deﬁne repeated projection by induction, as
[Equation]
Now consider any image X ∈ R d .
Its projection can be obtained by setting
[Equation]
Note that to avoid being exactly on the boundary, and thus retrieve information about the scales of θ ♠ and on which side of the boundary favors which label, we add a small noise ξ , to make sure Q does not lie exactly on V (which would lead to multiple solutions for the learning), but small enough so that the probabilities of the different label remain close to 0 .
1 (the equiprobable probability).
We acknowledge that images obtained this way may not be in [0 , 1] d , like the images of the MNIST dataset.
In general, one could search for points Q ∈ V ∩ [0 , 1] d .
Note that in theory, by Theorem(or a generalization of it), labeling random images in [0 , 1] d should sufﬁce.
However, in the case where V ∩ [0 , 1] d is empty (typically if no image in [0 , 1] is argued by model
θ
♠ s as realistically a 9), this procedure may require the labeling of signiﬁcantly more images to be successful.
I. A BRIEF THEORY OF DATA POISONING FOR LINEAR CLASSIFICATION
Using the efﬁcient poisoning data fabrication, we thus have a set of images ( Q , p ( Q )) , where p a ( Q ) is the probability assigned to image Q and label a .
This deﬁnes the following local loss for the strategic node:
[Equation]
where σ a ( θ s , Q ) =
exp( θ Tsa Q + θ sa 0 ) (cid:80) exp( θ Tsb Q + θ sb 0 ) is the probability that image Q has label a , according to the model θ s .
We acknowledge that such labelings of queries is unusual.
Evidently, in practice, an image may be labeled N times, and the number of labels N a it received can be set to be approximately
N a ≈ Np a ( Q ) .
It is noteworthy that the gradient of the loss function is then given by
[Equation]
where we deﬁned Q + (cid:44) (1 , Q ) (which allows to factor in the bias of the model.
This shows that ∇ θ s L s ( θ s , D s ) points systematically away from θ ♠ s , and thus that gradient descent will move towards
θ ♠ s .
In fact, if the set of images Q cover all dimensions (which occurs if there are Ω( d ) images, which is the case for 2,000 images, since d = 784 ), then gradient descent will always move the model in the direction of θ ♠ s , which will be the minimum.
Moreover, by overweighting each data ( Q , p ( Q )) by a factor α (as though the image Q was labeled α times), we can guarantee gradient-PAC* learning, which means that we will have θ ∗ s
≈ θ ♠ s , even in the personalized federated learning framework.
This shows why data poisoning should work in theory, with relatively few data injections.
Note that the number of other users does make learning harder.
Indeed, the gradient of the regularization R ( ρ, θ s ) at ρ = θ † s and θ s =
θ ♠ s is equal to 2 λ (cid:13)(cid:13) θ † s − θ ♠ s (cid:13)(cid:13) 2 .
As the number N − 1 of other users grows, we should expect this distance to grow roughly proportionally to N .
In order to make strategic user s robustly learn θ ♠ s , the norm of the gradient of the local loss L s at θ † s must be vastly larger than 2 λ (cid:13)(cid:13) θ † s − θ ♠ s (cid:13)(cid:13) 2 .
This means that the value of α (or, equivalently, the number of data injected in D s ) must also grow proportionally to N .
I. I NITIALIZATION OF THE LEARNING ALGORITHM
The convergence to the optimum is slow.
But given that the problem is convex, we focus here mostly on showing that the minimum is indeed a poisoned model.
To boost the convergence, we initialize our learning algorithm at a point close to what we expect to be the minimum, by taking this minimum and adding a Gaussian noise, and then we observe the convergence to this minimum.
J C IFAR -10 ON VGG 13-BN EXPERIMENTS
We considered VGG 13-BN, which was pretrained on cifar-10 by Phan (2021).
We now assume that 10 nodes are given part of the cifar-10 database, while a strategic node also joins to the personalized federated gradient descent algorithm.
The strategic node’s goal is to bias the global model towards a target model, which misclassiﬁes the cifar-10 data, by reclassifying 0 into 1, 1 into 2... and 9 into 0.
J. C OUNTER - GRADIENT ATTACK
We ﬁrst show the result of performing counter-gradient attack on the last layer of the neural network.
Essentially, images are now reduced to their vector embedding
[Figure]
and the last layer performs a simple linear classiﬁcation akin to the case of MNIST (see Appendix G)
[Figure]
J. R ECONSTRUCTING A MODEL ATTACK
Reconstructing an attack model whose effect is equivalent to the counter-gradient attack is identical to what was done in the case of MNIST (see Section 5.2)
[Figure]
J. R ECONSTRUCTING DATA POISONING
This last step is however nontrivial.
On one hand, we could simply use the attack model to label a large number of random images.
However, this solution would likely require a large sample com- plexity.
For a more efﬁcient data poisoning, we can construct vector embeddings on the indifference afﬁne subspace V , as was done for MNIST in Section 5.3.
This is what is shown below.
[Figure]
We acknowledge however that this does not quite correspond to data poisoning, as it requires reporting a vector embedding and its label, rather than an actual image and its label.
The challenge is then to reconstruct an image that has a given vector embedding.
We note that, while this is not a straightforward task in general, this has been shown to be at least somewhat possible for some neural networks, especially when they are designed to be interpretable (Zeiler & Fergus, 2014;
Wang et al., 2019c; Mai et al., 2019).