Learning versus Refutation in Noninteractive Local Differential Privacy
Abstract
We study two basic statistical tasks in non-interactive local differential privacy (LDP): learning and refutation : learning requires ﬁnding a concept that best ﬁts an unknown target function (from labelled samples drawn from a distribution), whereas refutation requires distinguishing between data distributions that are wellcorrelated with some concept in the class, versus distributions where the labels are random.
Our main result is a complete characterization of the sample complexity of agnostic PAC learning for non-interactive LDP protocols.
We show that the optimal sample complexity for any concept class is captured by the approximate  2 norm of a natural matrix associated with the class.
Combined with previous work, this gives an equivalence between agnostic learning and refutation in the agnostic setting
Introduction
We study two related basic statistical tasks, learning and refutation , in the setting of distributed data, and under strong privacy constraints.
For both tasks, we have an unknown distribution  on labeled data points in the universe U ⇥ {± 1 } , and we receive samples from  .
We are also given a concept class
C ✓ {± 1 } U , which, hopefully, is capable of capturing the labels given by  .
We deﬁne our two tasks as follows
• Learning requires ﬁnding a concept that best ﬁts  .
I
e., using the usual binary loss function L  ( h ) =
E ( a,b ) ⇠  [ I [ h ( a ) 6 = y ]] , the goal of agnostic learning with accuracy ↵ is to produce some h which, with probability 1   , satisﬁes L  ( h )  min c 2 C L  ( c ) + ↵
If an algorithm solves this problem for any distribution  , then we say it ( ↵ ,  ) -learns C agnostically. • Refutation requires distinguishing between data distributions  that are well correlated with some concept c 2 C , vs. data distributions where the labels are random.
I e., the goal of agnostic refutation with accuracy ↵ is to distinguish, with probability 1   , between the following two cases: (i) min c 2 C L  ( c )  12  ↵ versus (ii) for all h 2 {± 1 } U , L  ( h ) = 12 .
If an algorithm solves this problem for any distribution  , then we say it ( ↵ ,  ) -refutes
C agnostically.
The deﬁnition of agnostic learning above is classical.
Refutation is a more recent notion, and was studied by Kothari and Livni [2018] (and in a realizable setting by Vadhan [2017]), where it was 36th Neural Information Processing Systems (NeurIPS shown that computationally efﬁcient refutation is equivalent to computationally efﬁcient agnostic learning.
Refutation captures an extreme version of the problem of evaluating the choice of model in supervised learning, i.e., of estimating the best achievable loss min c 2 C L  ( c ) by the concept class
C .
While agnostic learning is well-deﬁned for any concept class, it is less meaningful when the best achievable loss is trivially large, which may be an indication that we need to choose a different model, i.e., a different concept class.
For this reason, ideally we would like our learning algorithm to also tell us what loss it is able to achieve.
Refutation is a more basic version of this problem, in which we merely want to distinguish data distributions for which our model is good from distributions with random labels, for which no model can achieve good results.
Certainly being able to solve the refutation problem is at least as hard as estimating min c 2 C L  ( c ) .
In this paper, we study learning and refutation in the model of non-interactive local differential privacy (LDP)
[Kasiviswanathan et al., 2008].
LDP applies in a distributed setting in which each data point represents one person, and, in order to protect privacy, the person retains ownership of their data point.
In particular, the data is never centrally collected, and, instead, the data owners communicate differentially private randomized message to a central server.
The differential privacy [Dwork et al., 2006] constraint ensures that the distribution on messages sent by one participant does not change dramatically if that participant’s data point is changed.
Thus, the central server or an outside observer cannot learn much about any particular data point, guaranteeing a strong form of privacy protection (as long as the privacy parameter is small enough).
Nevertheless, with enough participants, the combination of all private messages can reveal enough statistical information in aggregate in order to solve a statistical task, such as learning.
LDP is the model of choice of many industrial deployments of differential privacy [Erlingsson et al., 2014, Thakurta et al., 2017, Apple, 2016, Ding et al., 2017].
Here we focus on non-interactive LDP protocols, i.e., protocols in which each participant simultaneously sends a single message to the server.
Non-interactive protocols are much easier to implement than multi-round interactive protocols, particularly considering the large number of data points which are typically necessary for LDP to be useful.
Our main goal is to characterize, for any given concept class C , the sample complexity of learning and refutation under the constraints of non-interactive LDP.
Moreover, we aim to understand how these two problems are related to each other.
In many settings, it is trivial to take an algorithm for learning and use it to obtain an algorithm for refutation, by executing the learning algorithm for accuracy ↵ / 4 , and estimating the loss of the returned hypothesis within ↵ / 4 .
A converse of this simple reduction was established by Kothari and
Livni [2018], and by Vadhan [2017].
Unfortunately, neither of these reductions applies to the setting of non-interactive LDP, since they rely on interacting with the distribution  adaptively.
This leaves open the question of whether or not learning and refutation in the non-interactive LDP setting are equivalent tasks with respect to sample complexity.
In the forward (typically easier) direction, can non-interactive learning algorithms solve non-interactive refutation (by estimating the minimum loss achievable by the model) without a signiﬁcant increase in sample complexity, and conversely, can the reductions from refutation to learning from Kothari and Livni [2018] and Vadhan [2017] be extended to the setting of non-interactive protocols?
We note that, by the equivalence proved in Kasiviswanathan et al.
[2011] between LDP and the statistical queries (SQ) model of Kearns [1993], this also means that the relationship between the query complexity of non-adaptive SQ learning versus refutation is open.
Similarly, all our results extend to the non-adaptive SQ model.
Adaptive SQ learning has been characterized by Feldman [2017], and this in turn implies the same characterization for sequential LDP (LDP protocols in which each participant sends one message, which can depend on the messages of previous participants).
An overview of our main results follows.
The derivation of our results will be presented in Sectionfor the agnostic setting and in Section 3 for realizable.
See Appendix A for notation and standard deﬁnitions.
See Appendix E for a discussion of open problems.
1.1 Characterization of agnostic learning
Our ﬁrst theorem resolves the open problem for agnostic learning, by showing that non-interactive
LDP learning and refutation are equivalent (up to a logarithmic approximation) in the agnostic setting.
We do so by the following theorem, which gives a characterization of the sample complexity of both problems in terms of the approximate  2 norm of a natural matrix associated with the concept class
C ✓ {± 1 } U .
Theorem 1.
Let C ✓ {± 1 } U be a ﬁnite concept class with concept matrix
W 2 {± 1 } C ⇥ U , as given by Deﬁnition 3.
Let " > 0 , ↵ ,  2 (0 , 1 / 2] .
Then, to either (2 ↵ ,  ) -learn C agnostically, or (2 ↵ ,  ) -refute C agnostically under non-interactive " -LDP, it sufﬁces to have a sample of size
[Equation]
Conversely, for  = 12  ⌦ (1) , for some ↵ 0 = ⌦ ⇣ ↵ log(1 / ↵ ) ⌘ , the number of samples required to either ( ↵ 0 ,  ) -learn agnostically or ( ↵ 0 ,  ) -refute C agnostically under non-interactive " -LDP is at least
[Equation]
In Theorem 1, we denote by  2 ( W, ↵ ) the approximate  2 norm of the matrix W , i.e., the minimum  2 norm of a matrix that approximates W up to an additive ↵ entrywise.
(For a deﬁnition of the  2 norm, see Appendix A.)
The theorem shows that the sample complexity of both learning and refutation under non-interactive LDP can be characterized in terms of  2 ( W, ↵ ) .
Moreover, the sample complexities of both problems are equal, up to a factor O (log(1 / ↵ )) loss in the accuracy parameter, and a factor O (log |C| ) loss in the sample complexity.
The main new result in Theorem 1 is the lower bound on the sample complexity of learning.
The upper bound for both learning and refutation, as well as the lower bound for refutation were previously shown by Edmonds et al.
We close the gap by proving the lower bound for learning.
Combined with the previous results, this proves the equivalence of non-interactive LDP sample complexity for learning and refutation.
As in the previous proofs, we prove the lower bound for agnostic learning via the (dual formulation) of the approximate  2 norm.
In order to give a family of distributions that is hard against learning algorithms, we deﬁne a new difference matrix, D , associated with the concept class C , which is more suitable for the learning lower bound.
Then we show that  2 ( D, ↵ ) and  2 ( W, ↵ ) are approximately equal.
1.2 Characterization of realizable refutation
The results above do not apply to the realizable setting, in which the underlying distribution  on
U ⇥ {± 1 } is guaranteed to be labelled by a concept c 2 C , i.e., L  ( c ) = 0 .
In particular, the lower bounds we prove in terms of the approximate  2 norm utilize distributions that may not be realizable.
It is worth noting that some concept classes may require exponentially fewer samples to learn in the realizable case than to learn agnostically, as may be seen with Kearns [1993] and Feldman [2009].
While we are not able to characterize realizable learning, we give a characterization of a realizable analog of the refutation problem, and show that realizable learning is no harder than realizable refutation.
In our formulation of realizable refutation with accuracy ↵ , we are given samples from some distribution  over U ⇥ {± 1 } , and the goal is to distinguish the cases:
• min c 2 C L  ( c ) = 0 , i.e, some concept in C exactly gives the labels
under  ; • for all h 2 {± 1 } U , L  ( h )  ↵ .
The ↵ = 12 case is equivalent to the deﬁnition of refutation introduced by Vadhan [2017].
Daniely and Feldman [2018] showed that (for C closed under negation) the sample complexity of realizable learning under non-interactive LDP is bounded from below by the margin complexity of
C .
They left open the question whether one can prove a matching upper bound.
This question was resolved in the negative by Dagan and Feldman [2019].
The problem of characterizing the sample complexity of realizable learning under non-interactive LDP thus remains open.
In this work, we give a non-interactive LDP protocol which may be applied towards both realizable learning and realizable refutation.
This gives a sample complexity upper bound for these problems in terms of a new efﬁciently computable quantity ⌘ ( C , ↵ ) that we deﬁne.
This quantity combines elements of both the  2 norm and of margin complexity, and is sandwiched between them.
Further, we derive a lower bound for realizable refutation in terms of ⌘ ( C , ↵ ) , showing that our protocol is nearly optimal for realizable refutation, and that the sample complexity of realizable refutation is an upper bound on the sample complexity of realizable learning under non-interactive LDP.
Our main theorem for realizable learning is stated next.
See Section 3 for the deﬁnition of ⌘ ( C , ↵ ) .
Theorem 2.
Let C ✓ {± 1 } U be a ﬁnite concept class.
Let " > 0 , ↵ ,  2 (0 , 1 / 2] .
Then, to either (2 ↵ ,  ) -learn
C realizably, or (2 ↵ ,  ) -refute
C realizably under non-interactive " -LDP, it sufﬁces to have a sample of size
[Equation]
Conversely, for  = 12  ⌦ (1) , for some ↵ 0 = ⌦ ⇣
[Equation]
↵
[Equation]
⌘ , the number of samples required to ( ↵ 0 ,  ) -refute C realizably under non-interactive " -LDP is at least
[Equation]
Refutation versus Learning: Agnostic Case
As mentioned in the introduction, Edmonds et al.
[2019] (Theorems 24 and 25) gave sample complexity upper bounds for both agnostic learning and refutation for non-interactive LDP in terms of the approximate  2 , as well as a nearly tight lower bound for agnostic refutation.
However, it left open the question of lower bounds for agnostic learning under non-interactive LDP.
Our main theorem, stated next, resolves this by giving a nearly tight lower bound in terms of the approximate  2 norm of the natural matrix associated with C .
Theorem 1 thus follows by combining Theorems 24 andfrom Edmonds et al.
[2019] together with Theorem 4.[-
-]
Deﬁnition 3.
Let C ✓ {± 1 } U be a concept class.
The concept matrix W C 2 {± 1 } C ⇥ U of C is the matrix with entries given by w c,a = c ( a ) .
Theorem 4.
Let C ✓ {± 1 } U be a concept class with concept matrix W 2 {± 1 } C ⇥ U as given by
Deﬁnition 3.
Let " > 0 , ↵ ,  2 (0 , 1] .
Assume  = 12  ⌦ (1) .
Then, for some ↵ 0 = ⌦ ⇣ ↵ log(1 / ↵ ) ⌘ , under non-interactive " -LDP, the number of samples required to ( ↵ 0 ,  )
-learn C agnostically is at least
[Equation]
2.1 Difference matrix
Theorem 4 is given in terms of the concept matrix associated with the concept class; however, our proof of this result will focus instead on the difference matrix associated with the concept class, deﬁned below.
Deﬁnition 5.
The difference matrix of a concept class C : U ! {± 1 } is the matrix D 2 {± 1 } C 2 ⇥ U with entries given, for c, c2 C , a 2 U , by
[Equation]
The difference matrix is one of the key ideas that enables the proof of Theorem 4.
We will use a dual formulation of  2 ( D, ↵ ) to construct pairs of hard distributions for our lower bound, each pair corresponding to a pair of concepts c, c 0 2 C .
The structure of the difference matrix will help us ensure that no correct agnostic learning algorithm can output, with high probability, the same hypothesis for both distributions in a pair.
It is not apparent how to guarantee this property when working directly with the concept matrix W .
Nevertheless, the following lemma shows that  2 ( D, ↵ ) and  2 ( W, ↵ ) are essentially the same.
(See Appendix B for the proof.)
Lemma 6.
Let C be a concept class with concept matrix W 2 R C ⇥ U and difference matrix D[-
-]
R C 2 ⇥ U .
Then  2 ( D, ↵ )   2 ( W, ↵ ) .
Conversely,  2 ( W, ↵ )  2  2 ( D, ↵ / 2) + 1 , and if C is closed under negation then  2 ( W, ↵ )   2 ( D, ↵ ) .
The next Lemma is the same as Theorem 4, but with W replaced by the difference matrix D .
Theorem 4 is an immediate consequence of Lemma 7, together with Lemma 6.
Lemma 7.
Let C ✓ {± 1 } U be a concept class with concept matrix D 2 {± 1 } C ⇥ U as given by Deﬁnition 3.
Then Theorem 4 holds with W replaced by the difference matrix, D .
The rest of this section is devoted to the proof of Lemma 7.
2.2 KL-divergence bound
For a mechanism M : X n !
Z in the local model, and a probability distribution ⌫ on X n , we use
T M ( ⌫ ) to denote the distribution of the mechanism’s transcript when its input is sampled from ⌫ .
Lemma 8.
[Edmonds et al., 2019] Let " 2 (0 , 1] , and let M : X !
Z be a non-interactive " -LDP protocol.
Then, for families {  1 , . .
,  k } and { µ 1 , . .
. , µ k } of distributions on X , together with a distribution ⇡ over [ k ] , E
[Equation]
In matrix notation, deﬁne the matrix M 2 R [ k ] ⇥
X by m v,x = (  v ( x )  µ v ( x )) .
Then
[Equation]
Note that the statement of Lemma 8 is slightly different than the statement given in Edmonds et al. [2019], but the result as stated here is an immediate consequence of the original proof.
Our lower bound against agnostic learning will construct families {  1 , . .
,  k } and { µ 1 , . .
. , µ k } of distributions on
U ⇥ {± 1 } , as well as a distribution ⇡ over [ k ] .
The idea will be to construct these distributions
so that, when
M is an agnostic learner for C , then, for any ﬁxed v 2 [ k ] ,
[Equation]
where the ﬁrst inequality is just Pinsker’s inequality, and the second one will follow from our construction of  v and µ v .
Then the lower bound will follow by Lemma 8, as long as we can make sure that k M k ` 1 !
L 2 ( ⇡ ) is small.
2.3 Duality and Hard Distributions
For the construction of hard families of distributions, it will be convenient to make use of the following dual formulation, shown in Edmonds et al.
Lemma 9.
Let D 2 R C 2 ⇥
U be the difference matrix of a concept class C , as given by (1) .
Then,
[Equation]
For an arbitrary concept class C ✓ {± 1 } U , let U 2 R C 2 ⇥
U witness (2), so that
[Equation]
[Equation]
By normalizing U , we may assume, without loss of generality, that k U
k 1 = 1 .
Moreover, we can assume that, for any c, c 0 2 C ,
[Equation]
[Equation] Otherwise, U cannot achieve (2), since we can multiply the row of U indexed by ( c, c 0 ) by  1 , which increases D • U and does not change k U k 1 or  ⇤ 2 ( U ) .
We will consider the matrices U + , U  2 R C 2 ⇥
U with non-negative entries which satisfy U =
U +  U  , so that U +
and U  correspond to the positive and negative entries of U respectively.
We deﬁne the distribution ⇡ on C 2 by
[Equation]
Then, for c, c 0 2 C , consider the distribution  c,c 0 on U ⇥ {± 1 } given by
[Equation]
Similarly, let µ c,c 0 be the distribution on U ⇥ {± 1 } given by
[Equation]
Since U has unit ` 1 norm, the above distributions are well-deﬁned.
Note that  c,c 0 and µ c,c 0 have the same marginal on U which we denote  ( c,c 0 ) .
In particular,  ( c,c 0 ) ( a ) = | u ( c,c 0 ) ,a | ⇡ ( c,c 0 ) .
Meanwhile,  c,calways gives a the label b = sign ( u ( c,c 0 ) ,a ) , while µ c,c 0 always gives a the label b
=  sign ( u ( c,c 0 ) ,a ) .
It will be useful to have notation for one of these labelling functions, so deﬁne s c,c 0 : U ! {± 1 } by s c,c 0 ( a ) =
sign ( u ( c,c 0 ) ,a ) .
Consider the following relationship between U and the distributions we have constructed.
[Equation]
Note that
[Equation]
Similarly,
[Equation]
Hence,
D
[Equation]
Whenever C contains at least two distinct concepts,  2 ( D, ↵ ) > 0 , and then (3) implies D • U > ↵ .
By the equations above, this implies that, on average with respect to ( c, c 0 ) ⇠ ⇡ , the loss of c is greater by ↵ than the loss of c 0 on  c,c 0 .
Likewise, on average, the loss of c 0 is greater by ↵ than the loss of c on µ c,c 0 .
We will see later that, if we can obtain these properties in the worst case over all ( c, c 0 ) , rather than only on average, then no hypothesis can ﬁt both  c,c 0 and µ c,c 0 for any c, c 0 2 C .
The following section modiﬁes the distributions we have constructed so as to obtain the required properties.
2.4 Lower bound derivation
The following lemma gives a modiﬁcation of the hard distributions we constructed, giving us the properties we need for our lower bound.
The proof is deferred to Appendix C.
Lemma 10.
Let C be a concept class.
Let D be the matrix given by (1) .
Let U 2 R C 2 ⇥ U , k U k 1 = 1 , satisfy (3) .
Then there exist probability distributions
e  c,c 0 and e µ c,c 0 over U ⇥ {± 1 } , and a distribution e ⇡ over C 2 such that:
For all ( c, c 0 ) in the support of
[Equation]
For all ( c, c 0 ) in the support of
[Equation]
The matrix
[Equation]
entries
[Equation]
We also want to bound the operator norm, which appears in Lemma 8, in terms of
⇤ 2 ( U ) .
To do so, we use the following lemma from Edmonds et al.
Lemma 11 ([Edmonds et al., 2019]) .
Let U and M be k ⇥ T matrices, and let ⇡ be a probability distribution on [ k ] such that, for any i 2 [ k ] , j 2 [ T ] , we have u
i,j = ⇡ ( i ) m i,j .
Then there exists a probability distribution b ⇡ on [ k ] , with support contained in the support of ⇡ , such that
[Equation]
Recall that we also want to obtain a lower bound on d TV ( T M (  nc,c 0 )
k T M ( µ nc,c 0 ))
when M is a learning algorithm for
C .
For this purpose, we apply the following lemma, whose proof is also deferred to Appendix C.
The main observation in the proof is, for any hypothesis h : C ! {± 1 } , and any distributions  and µ satisfying the conditions of the lemma, we have L  ( h ) + L µ ( h )
= 1 .
Lemma 12.
Let  and µ be distributions on U ⇥ {± 1 } .
Assume that  and µ have the same marginal on
U .
Also assume that  is labelled by some s : U ! {± 1 } while µ is labelled by  s .
Finally, assume that for some c, c2 C , L µ ( c 0 )  L µ ( c ) > ↵ . If h
[Equation]
satisﬁes L  ( h )  L  ( c
[Equation]
+ ↵ / 4 , then L µ ( h ) >
L µ ( c ) + 3 ↵ / 4 .
Hence, if M is an algorithm which ( ↵ / 4 ,  ) -learns C from n samples, then d TV ( M (  n ) , M ( µ n ))  1  2  .
Finally, with these results at our disposal, we may obtain the lower bound of Lemma 7.
Proof of Lemma 7.
Let U 2 R C 2 ⇥ U , k U k 1 = 1 , satisfy (3).
Let e ⇡ , together with e  c,c 0 and e µ c,c 0 be the distributions guaranteed to exist by Lemma 10 and let e
U 2 R C 2 ⇥ U be the corresponding matrix with entries
[Equation]
⌘ Let M be the matrix with entries m ( c,c 0 ) ,a = e u ( c,c 0 ) ,a / e ⇡ ( c, c 0 ) .
By Lemma 11, there exists some distribution ˆ ⇡ with support contained in that of e ⇡ such that
[Equation]
where the last inequality follows from Lemma 10.
Combining Lemma 8 with the dual formulation (3) gives
[Equation]
Now let
[Equation]
where the last inequality is by Lemma 10.
By Lemma 12, if M ( ↵ 0 ,  ) -learns C for some  = 12  ⌦ (1) , then
E ( c,c 0 )
⇠ e ⇡ h
D KL [Equation] M [Equation]
e  nc
[Equation]
c 0 ) k [Equation] e µ nc
[Equation]
c [Equation] [Equation] i = ⌦ (1) .
This implies n = ⌦ ✓ ⇣  [Equation] D, ↵ [Equation]
"↵ ⌘ ◆ , as was to be proved.
Refutation versus Learning: Realizable Case
In this section, we present our algorithm for realizable learning and refutation for non-interactive
LDP.
For a concept class C : U ! {± 1 } , we deﬁne a quantity ⌘ ( C , ↵ ) and argue that it gives an upper bound on the sample complexity for realizable learning of
C .
Deﬁnition 13.
Let C : U ! {± 1 } be a concept class.
Let
[Equation]
Let
[Equation]
where 1 T is the all-ones row vector indexed over C , so that
[Equation]
T is the matrix obtained by shifting each row c of W in each entry by ✓ c .
Then deﬁne
[Equation]
The idea is that each row of W deﬁnes a statistical query corresponding to a concept, q c ( a, b ) =
w c, ( a,b ) .
The statistical query corresponding to the true concept that was used to label the data will have a small value, whereas any query corresponding to a concept with large loss will have a large value.
The next theorem formalizes this argument.
Theorem 14.
Let C ✓ {± 1 } U be a concept class.
Let " > 0 , ↵ ,  2 (0 , 1] .
Then there exists an " -LDP mechanism which may be used to both (3 ↵ ,  ) -learn
C realizably and (3 ↵ ,  ) -refute C realizably with n samples, where
[Equation]
Proof.
As per Deﬁnition 13, let f
W 2 K 0 C be the matrix that witnesses ⌘ ( C , ↵ ) and let W 2 K C and ✓ 2 R C be the matrix and vector which witness f
W 2 K 0 C .
If we can answer the statistical queries given by f
W , then we can answer the queries given by W with the same accuracy by subtracting ✓ c from the answer to the query for concept c .
By the deﬁnition of W , if, for some c 2 C ,  is supported on on those ( a, b ) 2 U ⇥ {± 1 } which satisfy c ( a )
= b , then the value of the query corresponding to c is bounded as
[Equation]
Meanwhile, for an arbitrary distribution  on U ⇥ {± 1 } , the value of the query corresponding to c 2 C may be bounded as
[Equation]
In particular, if L  ( c )  3 ↵ , then
[Equation]
It follows that, by approximating the statistical queries given by W with worst-case error ↵ 4 , we can distinguish the case where  agrees with some c 2 C from the case where, for all concepts c 2 C , L  ( c )  3 ↵ .
In the former case, returning some c 0 2 C where our estimate of
[Equation] strictly less than 2 ↵ guarantees L  ( c ) < 3 ↵ .
To complete the proof, it sufﬁces to apply the upper bound from Edmonds et al.
[2019] which says that, to answer the collection of statistical queries given by f
W under non-interactive " -LDP, with accuracy ↵ / 4 and probability of failure at most  , the number of samples required is at most
[Equation]
3.1 Lower bound
Our lower bound will follow a similar strategy as in the agnostic case.
However, our construction of hard distributions will be tailored to ⌘ ( C , ↵ ) and its dual.
3.1.1 Duality
We will again use convex duality in our lower bound.
We will express ⌘ ( C , ↵ ) as a maximum over dual matrices U , and we will use an optimal U to construct ‘hard distributions’ for realizable refutation.
To this end, consider the following duality lemma, proved in Appendix D.
Lemma 15.
For any concept class C ✓ {± 1 } U and any ↵ ,
[Equation]
where we deﬁne
[Equation]
3.1.2 Hard distributions
Let U 2 R C ⇥
( U ⇥ {± 1 } ) witness (11).
By normalizing, we may assume without loss of generality that k U
k 1 = 1 .
We will consider the matrices U + , U  2 R m ⇥ N with non-negative entries which satisfy
U =
U +  U
so that U +
and U  correspond to the positive and negative entries of U respectively.
We deﬁne the distribution ⇡ on C given by ⇡ ( c ) =
P
[Equation]
{±
[Equation]
c, ( a
[Equation]
Then, for each c 2 C , let  c and µ c be the distributions on U ⇥ {± 1 } given by
[Equation]
and
[Equation]
Since the rows of U each sum to zero and have unit ` 1 norm, the distributions  c and µ c are welldeﬁned.
Moreover, since u c, ( a,  c ( a ))  0 for all c 2 C , a 2 U , the only negative entries of U are those of the form u c, ( x,c ( x )) .
This implies that the distribution µ c always labels samples a 2 U by c ( a ) .
3.1.3 Warm-up: single-concept case
Consider the case where C consists of a single concept c .
Since ⌘ ( C , ↵ ) > 0 , then (11) implies
[Equation] [Equation]
Hence,
P
[Equation]
Using [Equation] and rearranging, this gives
[Equation]
In other words, if we can distinguish a distribution on U ⇥ {± 1 } which labels samples according to c from one which disagrees with c with probability greater than 2 ↵ 1+ ↵ , then we can distinguish between  c and µ c .
3.1.4 General case
Appendix D.2 generalizes the lower bound of the previous section to the general case where the concept class is not restricted to a single concept.
The ﬁrst issue which needs to be addressed in the general case is that (12), rather than holding in worst case over all concepts, holds on average for a concept c 2 C drawn from the distribution ⇡ .
This issue is handled by applying the binning result of Lemma 19.
The second issue which needs to be addressed is that, while each c 2 C is guaranteed not to ﬁt the corresponding distribution  c , as with (13), it may hold that some other h : U ! {± 1 } has small loss on  c .
This is remedied by mixing a distribution  c which agrees with c into the distribution  c .
This guarantees that every h : U ! {± 1 } has large loss on  c .
Altogether, we obtain the lower bound of Theorem 1.