SQ Lower Bounds for Learning Single Neurons with Massart Noise
Abstract
We study the problem of PAC learning a single neuron in the presence of Massart noise.
Speciﬁcally, for a known activation function f : R → R , the learner is given access to labeled examples (
x , y ) ∈ R d × R , where the marginal distribution of x is arbitrary and the corresponding label y is a Massart corruption of f ( (cid:104) w , x (cid:105) ) .
The goal of the learner is to output a hypothesis h : R d → R with small squared loss.
For a range of activation functions, including ReLUs, we establish superpolynomial Statistical Query (SQ) lower bounds for this learning problem.
In more detail, we prove that no efﬁcient SQ algorithm can approximate the optimal error within any constant factor.
Our main technical contribution is a novel SQhard construction for learning {± 1 } -weight
Massart halfspaces on the Boolean hypercube that is interesting on its own right.
Introduction
The success of deep learning has served as a motivation for understanding the complexity of learning simple classes of neural networks.
Here we study arguably the simplest possible setting of learning a single neuron, i.e., a real-valued function of the form x (cid:55)→ f ( (cid:104) w , x (cid:105) ) , where w is the weight vector of parameters and f : R (cid:55)→ R is a non-linear and monotone activation.
The underlying learning problem is the following:
Given i.i.d.
samples from a distribution D on ( x , y ) , where x ∈ R d is the example and y ∈ R is the corresponding label, the goal is to learn the target function in L 22 -loss.
That is, the objective of the learner is to output a hypothesis h : R d (cid:55)→ R such that E ( x ,y )
∼D
[( h ( x ) − y ) 2 ] is as small as possible, compared to the optimal loss value OPT :=
min w ∈ R d E ( x ,y ) ∼D [( f ( (cid:104) w , x (cid:105) ) − y ) 2 ] .
A learning algorithm in this context is called proper if the hypothesis h is restricted to be of the form h (cid:98) w ( x ) =
f ( (cid:104) (cid:98) w , x (cid:105) ) , for some (cid:98) w ∈ R d .
One of the most popular activations is the ReLU function, corresponding to f ( u )
= ReLU( u ) def = max { 0 , u } .
In this work, we study the complexity of improperly learning single neurons, where the marginal distribution on examples is ﬁxed but arbitrary and the hypothesis h is allowed to be any efﬁciently computable function.
In the realizable case, i.e., when the labels are consistent with a function in the target concept class, the above learning problem is known to be efﬁciently solvable for various activation functions.
A line of work, see, e.g., [KS09, Sol17, YS20] and references therein, has shown that simple algorithms like gradient-descent efﬁciently converge to an optimal solution (in some cases under assumptions on the marginal distribution on examples).
On the other hand, in the adversarial label noise (aka agnostic) model, known hardness results [Dan16, DKMR22] rule out efﬁcient constant factor approximations to the optimal loss for a range of activations including ReLUs.
The aforementioned negative results for label agnostic learning motivate the study of weaker corruption models, where non-trivial efﬁcient learning algorithms may still be possible.
A natural class of such models — that may be more realistic in some practical applications — are semi-random noise models, involving a combination of adversarial choices and random choices.
Here we focus on the Massart (or bounded) noise model [MN06], a classical semi-random model ﬁrst deﬁned in the context of binary classiﬁcation (see [Slo88] for an equivalent noise model).
Intuitively, in the Massart model, an adversary has control over a (uniformly) random η < 1 /fraction of the labels (see Deﬁnition 1.1).
In the context of binary classiﬁcation, [DGT19] gave the ﬁrst non-trivial learning algorithm for halfspaces in this model (see also [CKMY20]).
Subsequent work [DK20, NT22] provided evidence that the error guarantee of the latter algorithm is essentially best possible in the Statistical Query (SQ) model [Kea98].
To state our results, we formally deﬁne the following natural generalization of the model for real-valued functions (see [CKMY21, DPT21]).
Deﬁnition 1.1 (Massart Noise Model) .
Let G be a concept class of real-valued functions over R d , D x be a ﬁxed distribution over R d , and 0 < η < 1 / 2 .
Fix an unknown function g ∈ G .
The noiseless distribution D (corresponding to g ) is the distribution on labeled examples ( X , Y ) , supported on R d × R , where
X ∼ D x and Y = g ( X ) .
An η -Massart distribution , D Mass η , is a distribution on labeled examples ( X , Y (cid:48) ) , supported on R d × R , such that for ( X , Y (cid:48) ) ∼ D Mass η we have that (i)
X ∼ D x , and (ii) for all x ∈ R
d
it holds that Pr ( X ,Y (cid:48) )
∼D
[Equation]
Given sample access to the η -Massart distribution D Mass η , corresponding to an unknown g ∈ G , the goal is to output a hypothesis h : R d (cid:55)→ R such that L 2 ( h ; D Mass η ) :
=
E ( X ,Y (cid:48) )
∼D Mass η [( Y (cid:48) − h
( X )) 2 ] is small.
Let OPT Mass :=
inf g ∈G E ( X ,Y (cid:48) )
∼D Mass η [( Y (cid:48) − g ( X )) 2 ] denote the optimal squared error.
We will say that a learning algorithm is α -approximate if it outputs an h : R d (cid:55)→ R that with high probability satisﬁes L 2 ( h ; D Mass η ) ≤ α ( d ) · OPT Mass .
We say that a learner is a constant factor approximation if α = O (1) .
We focus on the concept class of single neurons: for an activation f : R (cid:55)→ R , we will denote by C f def = { c w : R d (cid:55)→ R | c w ( x ) =
f ( (cid:104) w , x (cid:105) ) , w ∈ R d } .
In Deﬁnition 1.1, the Massart adversary corrupts each label independently with probability at most η .
Even though this noise model might appear innocuous, the fact that the corruption probability is unknown to the learner makes the design of efﬁcient Massart learners challenging.
The Massart model has been extensively studied in the context of binary classiﬁcation [Slo88, RS94, Slo96,
ABHU15, ABHZ16, DGT19, DKTZ20, CKMY20, DKT21] and, more recently, for learning realvalued functions [CKMY21, DPT21].
For PAC learning halfspaces with Massart noise, there is evidence that even approximate learning is computationally hard in the SQ model [DK20, NT22].
In sharp contrast, our understanding of the possibilities and limitations of Massart learning well-behaved real-valued functions (including
ReLUs and other Lipschitz monotone activations) remains limited.
On the positive side, recent work developed the ﬁrst efﬁcient learners for linear regression [CKMY21, DPT21] and ReLU regression [DPT21] with Massart noise.
We note that the ReLU regression algorithm of [DPT21] requires a certain anti-concentration condition on the distribution D x of examples, which is crucial for its performance guarantees.
In fact, without such an assumption, no non-trivial upper bound is known for ReLUs (or other non-linear activations).
This discussion prompts the following question:
Is there an efﬁcient O (1) -approximate learner for distribution-free learning of a single neuron with Massart noise?
For the important case of ReLU activations, [DPT21] conjectured that the distribution-independent
PAC learning problem is intractable.
As the main contribution of this paper, we provide strong evidence towards this conjecture, by establishing super-polynomial lower bounds in the Statistical
Query (SQ) model — a restricted but powerful family of algorithms.
Speciﬁcally, we show that no efﬁcient SQ algorithm can achieve any constant factor approximation.
Moreover, our SQ-hardness result is not speciﬁc to ReLUs, but generalizes to a broad class of non-linear activation functions.
1.1 Our Results
In this work, we give strong evidence that the problem of learning single neurons with Massart noise does not admit any constant factor approximation.
Speciﬁcally, we show that any efﬁcient SQ algorithm cannot achieve a constant factor approximation.
In fact, the hardness gap that we establish is super-constant, scaling with the dimensionality of the problem.
Instead of directly accessing samples, SQ algorithms [Kea98] are only to adaptively query expectations of bounded functions of the underlying distribution up to some tolerance (see Section 2).
The class of SQ algorithms is fairly broad: a wide range of known algorithmic techniques in machine learning are known to be implementable in the SQ model [FGR + 17].
For the important class of ReLU activations, our main result is the following:
Theorem 1.2 (SQ Hardness of Massart Learning ReLUs) .
Any SQ algorithm that learns a single neuron with ReLU activation on R d , in the presence of Massart noise with η = 1 / 3 , to squared error better than 1 / poly(log( d )) requires either queries of accuracy better than 2 − (log d ) c 1 or at least 2 (log d ) c 1 statistical queries, for some constant c 1 > 1 .
This holds even if the optimal squared error is at most 2 − (log d ) c 2 for some 0 < c 2 < 1 , and the total weight of the neuron is poly( d ) .
Theorem 1.2 rules out the existence of efﬁcient SQ algorithms (i.e., using polynomially many queries of inverse polynomial accuracy) with approximation ratio 2 (log d ) c for some 0 < c < 1 .
It therefore a fortiori rules out any constant factor approximate SQ learner.
We note that the SQ-hardness result of Theorem 1.2 does not require the linearity of the ReLU (on positive inputs); a similar result can be shown for a broader class of activation functions.
Speciﬁcally, we can generalize our SQ-hardness result to any activation f of the form f ( u ) = 0 , u < 0 , and f ( u ) =
φ ( u ) otherwise, where φ is any injective function.
Theorem 1.2 establishes SQ-hardness of learning single neurons under the Massart noise notion of Deﬁnition 1.1.
We note that for learning real-valued functions, one can consider other natural deﬁnitions of “Massart noise”.
Speciﬁcally, Deﬁnition 1.1 considers an L 0 -perturbation (the adversary is allowed to arbitrarily corrupt a random η -fraction of the labels).
Another natural deﬁnition considers
L 2 -perturbations, as stated below.
Deﬁnition 1.3 ( L 2 -Massart Noise Model) .
Let G be a concept class of real-valued functions over R d , D x be a ﬁxed distribution over R d , and η > 0 .
Fix an unknown function g ∈ G .
An η - L 2 -Massart distribution , D Mass − L2 η , is a distribution on labeled examples ( X , Y ) , supported on R d × R , such that for ( X , Y ) ∼ D Mass − L2 η we have that (i)
X ∼ D x , and (ii) for all x ∈ R
d it holds that E ( X ,Y )
∼D Mass − L2 η
[( Y
[Equation]
g
[Equation]
2 | X = x ] ≤ 4 η.
We will use
OPT
[Equation] [Equation]
) 2 ] to denote the optimal squared error.
Note that for {± 1 } labels, with noise rate η < 1 / 2 , the above model generalizes the standard
Massart model (for binary classiﬁcation) with the same noise rate η .
For this noise model, we establish SQ-hardness for the following general family of non-linear activations (including ReLUs):
Deﬁnition 1.4 (Fast Convergent Activation) .
We say that a function f : R (cid:55)→ R is a fast-convergent activation if either g ( t ) :
= f ( t ) or g ( t ) :
= f ( − t ) satisﬁes the following: (i)
lim
t →−∞ g ( t ) exists.
(ii) For t < 0 with absolute value sufﬁciently large, | g ( t ) − g ( −∞ ) | = 1 / poly( | t | ) .
Intuitively, the second condition above requires that the function converges to its limit at inverse polynomial rate.
Without loss of generality, we consider activation which converge on the negative side.
For such an activation f , let f − :=
f ( −∞ ) and c + be a constant such that f ( c + ) (cid:54) =
f − .
Our proof technique establishing Theorem 1.2 is quite robust and can be adapted to L 2 -Massart noise under fast convergent activations.
Our main result in this context is the following:
Theorem 1.5 (SQ Hardness of L 2 -Massart Learning) .
Let f : R
(cid:55)→ R be a fast convergent activation.
Any SQ algorithm that learns a single neuron with activation f on R d , in the presence of η - L 2Massart noise with η = 2( f ( c + ) − f − ) 2, to squared error better than 1 / poly(log( d )) requires either queries of accuracy better than 2 − (log d ) c 1 or at least 2 (log d ) c 1 statistical queries, for some constant c 1 > 1 .
This holds even if the optimal squared error is at most 2 − (log d ) c 2 for some 0 < c 2 < 1 , and the total weight of the neuron is poly( d ) .
Interestingly, the key ingredient for our aforementioned SQ-hardness results for real-valued functions is a new SQ-hardness construction for low-weight halfspaces (i.e., neurons with a sign activation) on the Boolean hypercube.
In this context, we prove:
Theorem 1.6 (SQ Hardness for Low-weight Massart Halfspaces on { 0 , 1 } d ) .
Any SQ algorithm that learns {± 1 } -weight halfspaces on { 0 , 1 } d , in the presence of Massart noise with η = 1 / 3 , to 0error better than 1 / poly(log( d )) requires either queries of accuracy better than 2 − (log d ) c 1 or at least 2 (log d ) c 1 statistical queries, for some constant c 1 > 1 .
This holds even if the optimal 0-1 error is at most 2 − (log d ) c 2 for some 0 < c 2 < 1 .
Theorem 1.6 rules out any efﬁcient polynomial (relative) approximation for {± 1 } -weight halfspaces on the hypercube.
This is the ﬁrst hardness result for approximate learning of Boolean Massart halfspaces.
Prior work either obtained SQ-hardness of exact learning [CKMY20] or was inherently applicable to halfspaces on R d [DK20, NT22].
A number of learning problems involving halfspaces are computationally easy when the weights are small integers (aka in the “large margin” case) and computationally hard for arbitrary weights.
The conceptual message of Theorem 1.6 is that the Massart halfspace learning problem is SQ-hard due to the combinatorial nature of the problem (and not due to the magnitude of the weights).
This addresses an open problem of [Blu03] regarding the complexity of Massart learning simple halfspaces.
1.2 Technical Overview
We start by describing the proof of Theorem 1.6.
Our SQ lower bound for learning Boolean Massart halfspaces requires a number of novel ideas.
Our starting point is the construction of [DK20] that proves a similar lower bound in the continuous setting.
They begin by producing a one-dimensional construction of a Massart PTF whose distributions conditional on y = 1 and y = − 1 approximately match many moments.
Using techniques from [DKS17], they show that by embedding this onedimensional construction into higher dimensions, they can produce d -dimensional instances of
Massart polynomial threshold functions (PTFs) that are SQ-hard to learn.
They then further embed these instances via the Veronese embedding to produce SQ-hard LTF instances (essentially using the fact that a PTF in x is an LTF in the low-degree monomials of x ).
Our proof adapts this general idea to the discrete setting.
The ﬁrst obstacle is developing an appropriate analogue of the one-dimensional construction.
The construction from [DK20] uses the fact that a discrete Gaussian nearly matches moments with a standard Gaussian; thus, making the conditional distributions of x mixtures of discrete Gaussians ensures that the moment matching condition is satisﬁed.
By carefully picking this mixture, they ensure that the conditional distributions have no overlap for | x | small (thus ensuring a small value of OPT ), but that the y = 1 case is always more likely for | x | sufﬁciently large.
This construction does not work in our case, as we need our one-dimensional construction to be discrete.
Our basic idea is to begin by noting that the binomial distribution conditioned on x beingmod s approximately matches many moments with the full binomial.
As a ﬁrst attempt, we let y =
− 1 if x ≡ 0 mod s and y = 1 otherwise.
This matches many moments with the binomial, but alternates between y = 1 and y =
− 1 many times, and thus cannot be considered to be a low-degree
PTF with Massart noise.
To ﬁx this, we need to modify our distributions so that: (i) Conditioned on any x far from n/ 2 , y is more likely to be 1 than − 1 , (ii) the two distributions conditioning on y = 1 and y = − 1 have little overlap, and (iii) each conditional distribution approximately matches moments with the full binomial.
We can ﬁx (i) at the cost of (ii) by replacing the conditional distribution on y = 1 with the full binomial distribution.
As long as the prior probability of y =exceeds that of y = − 1 by enough, even for x ≡ 0 mod s , y = 1 will be more likely than y = − 1 .
Unfortunately, the conditional distributions now have too much overlap.
We can address this by moving the mass in the y = 1 conditional off of the points with x ≡ 0 mod s and | x − n/ 2 | small.
Importantly, we must ﬁnd a way to do this without destroying property (iii).
To that end, we show that there is a way to move mass from each of these points x and redistribute it to nearby points in such a way so as to not affect any of the low-order moments (see Lemma 3.8).
By doing this to each x ≡ 0 mod s with | x − n/ 2 | small, we get our ﬁnal construction.
We also need to modify the method by which we embed the one-dimensional construction into higher dimensions in order to get the family of SQ-hard PTF instances.
This construction must differ from previous constructions, as our family of distributions will be discrete and not Gaussian-like as in [DKS17].
Fortunately, we can leverage the recent technique of [DKS22], embedding our low-dimensional construction as a junta.
In particular, signiﬁcant difference with the Gaussian case is in the way we embed the low-dimensional distribution as a higher-dimensional one.
In the Gaussian case, we simply take the distribution to be Gaussian in independent directions.
In our discrete setting, we begin by embedding into a moderate dimensional hypercube by taking the unique symmetric distribution, where our one-dimensional distribution over some subset S is the distribution over (cid:80) i ∈
S X i .
We note that this distribution will approximately match low-degree moments with the uniform distribution over the hypercube.
We then embed this distribution into a higher-dimensional hypercube as a random junta.
As an application of the above general recipe for SQ lower bounds for discrete distributions, we note that the hard instances we construct for learning Boolean halfspaces with Massart noise, also (with a slight change of variables) produce hard instances for ReLUs (and other activations).
In particular, in our hard instance for PTFs, the optimal classiﬁer f is given by f ( x ) = − 1 if x S ≡mod s and | x S − n/ 2 | < ds/ 2 , and 1 otherwise, where x S is the sum over the coordinates of x in some particular subset S .
We note that the function (1 − f ( x )) / 2 , which is equal to 1 if x S ≡mod s and | x S − n/ 2 | < ds/ 2 and 0 otherwise, can be written as ReLU( p ( x )) for some degree
O ( d ) polynomial p , where p ( x ) = 1 for x S ≡ 0 mod s and | x S − n/ 2 | < ds/ 2 , and p ( x ) ≤otherwise.
By replacing x by its Vernonese embedding as before, we can produce hard instances of
ReLU functions with Massart noise.
Preliminaries
Notation For n ∈ Z + , we denote [ n ] def = { 1 , . . . , n } .
For two distributions p, q over a probability space Ω , let d T V ( p, q )
=
sup S ⊆ Ω | p ( S )
− q ( S ) | denote the total variation distance between p and q .
We use Pr [ E ] and I [ E ] for the probability and the indicator of event E .
For a real random variable
X , we use E [ X ] , Var [ X ] to denote the expectation and variance of X , respectively.
For n ∈ Z + and 0 ≤ p ≤ 1 , we use Bin( n, p ) to denote the Binomial distribution with parameters n and p .
Throughout this article, we will use capital letters (e.g., X, X ) to denote random variables and random vectors, and small letters (e.g, x, x ) to denote corresponding values.
Statistical Query Algorithms We will use the framework of Statistical Query (SQ) algorithms for problems over distributions [FGR + 17].
We require the following standard deﬁnition.
Deﬁnition 2.1 (Decision/Testing Problem over Distributions) .
Let D be a distribution and D be a family of distributions over R M .
We denote by B ( D , D ) the decision (or hypothesis testing) problem in which the input distribution D (cid:48) is promised to satisfy either (a) D (cid:48)
=
D or (b) D (cid:48)
∈ D , and the goal of the algorithm is to distinguish between these two cases.
We deﬁne SQ algorithms as algorithms that do not have direct access to samples from the distribution, but instead have access to an SQ oracle.
We will consider the following standard oracle.
Deﬁnition 2.2 ( STAT Oracle) .
Let D be a distribution on R M .
A Statistical Query (SQ) is a bounded function f : R M
→ [ − 1 , 1] .
For τ > 0 , the STAT( τ ) oracle responds to the query f with a value v such that | v − E X ∼ D [ f ( X )] | ≤ τ .
We call τ the tolerance of the statistical query.
A Statistical
Query (SQ) algorithm is an algorithm whose objective is to learn some information about an unknown distribution D by making adaptive calls to the corresponding STAT( τ ) oracle.
SQ Hardness Construction for Supervised Learning 3.1 Generic SQ Lower Bound Construction
We start with some basic deﬁnitions.
Let U M be the uniform distribution over { 0 , 1 } M .
For a subset
T ⊆
[ M ] and x ∈ { 0 , 1 } M , we denote χ T ( x ) =
( − 1) (cid:80) i ∈ T x i .
For a distribution P over { 0 , 1 } M , let (cid:98) P ( T ) =
E X ∼ P [ χ T ( X )] .
We will require the orthogonal polynomials under the binomial distribution.
Deﬁnition 3.1 (Kravchuk Polynomial [Sze89]) .
For k, m, x ∈ Z + with 0 ≤ k, x ≤ m , the
Kravchuk polynomial K k ( x ; m ) is the univariate degree- k polynomial in x deﬁned by K k ( x ; m ) :
= (cid:80)
T ⊆ [ m ] , | T | = k χ T ( y ) =
(cid:80) kj =
( − 1) j (cid:0) xj (cid:1)(cid:0) m − x k − j (cid:1) , where y has x 1’s and m − x 0’s.
The following distribution family that is the basis of our discrete SQ lower bound construction.
Deﬁnition 3.2 (High-Dimensional Hidden Junta Distribution) .
Let m, M ∈ Z + with m < M .
For a distribution A on [ m ] ∪ { 0 } with probability mass function (pmf) A ( x ) and a subset S ⊆
[ M ] with | S | = m , consider the probability distribution over { 0 , 1 } M , denoted by P AS , such that for
X ∼ P
AS the distribution ( X i )
i
(cid:54)∈ S is the uniform distribution on its support and the distribution ( X i ) i ∈ S is symmetric with (cid:80) i ∈
S X i distributed according to A .
Speciﬁcally, P AS is given by the pmf
P AS ( x ) =
− M + m A (cid:0)(cid:80) i ∈
S x i (cid:1) (cid:0)
[Equation]
The following condition describes the approximate moment-matching property of the desired distribution A with the Binomial distribution.
Condition 3.3.
Let k, m ∈ Z + with k < m and ν > 0 .
The distribution A on [ m ] ∪ { 0 } is
such that | E X ∼ A
[ K t ( X ; m )] | ≤ ν , for all 1 ≤ t ≤ k .
We now deﬁne the hypothesis testing and learning problem which will be used throughout this paper:
Deﬁnition 3.4 (Hidden Junta Binary Testing Problem) .
Fix a (cid:54) =
b ∈ R .
Let A and B be distributions on [ m ] ∪ { 0 } satisfying Condition 3.3 with parameters k ∈ Z + and ν ∈ R + , and let p ∈
For M ∈ Z + , M > m , and a subset S ⊆
[ M ] with | S | = m , deﬁne the distribution P A,B,pS,a,b on { 0 , 1 } M × { a, b } that returns a sample from ( P AS , a ) with probability p and a sample from ( P BS , b ) with probability 1 − p .
In the ( A, B, a, b, M ) -Hidden
Junta Testing Problem, one is given access to a distribution D so that either H 0 : D = U pM , where for ( X , Y ) ∼ U pM we have that X is a uniform random element of { 0 , 1 } M , and Y is independently a with probability p and b with probability 1 − p .
H 1 : D is given by P A,B,pS,a,b for some subset S ⊆
[ M ] with | S | = m .
One is then asked to distinguish between H 0 and H 1 .
Note that this is just the hypothesis testing problem B ( D , D ) with D = U pM and D
= { P A,B,pS,a,b } .
Proposition 3.5 (Generic Discrete SQ Lower Bound) .
Let m, M ∈ Z + with M > m .
Let A, B be distributions on [ m ] ∪ { 0 } satisfying Condition 3.3.
Let τ ≥ kν 2 + 2 − k
( χ 2 ( A, Bin( m, 1 / 2))
+ χ 2 ( B, Bin( m, 1 / 2))) .
Any SQ algorithm that solves the testing problem of Deﬁnition 3.4 with probability at least 2/3 must either make queries of accuracy better than √ 2 τ or must make at least 2 Ω( m ) τ/ ( χ 2 ( A, Bin( m, 1 / 2))
+ χ 2 ( B, Bin( m, 1 / 2))) statistical queries.
We give a proof sketch here and defer the full proof to Appendix B. We ﬁrst pick a collection C of s = 2 Ω( m ) subsets S ⊆
[ M ] of size m whose pairwise intersections are small.
For any S, S (cid:48) ∈ C , we write the correlation between P A,B,pS,a,b , P A,B,pS (cid:48) ,a,b as a linear combination of the correlation between
P AS , P AS (cid:48) and the correlation between P BS , P BS (cid:48) .
Since we can directly obtain an upper bound for the correlation between the hidden junta distributions, applying existing techniques will yield our result.
3.2 Construction of Univariate Moment-Matching Distributions
Here we give the construction of our approximate moment-matching distributions.
For convenience, we use the “expectation” and “moments” for the unnormalized measure without clariﬁcation.
The main result of this section is captured in the following proposition.
Proposition 3.6.
Let d, k, s, m ∈ Z + and ζ ∈ (0 , 1 / 2) such that: (i) s ≥ ω ( k 4 ) , (ii) k < m/ 2 , (iii) ds ≥ Ω( (cid:112) m log(1 /ζ )) , and (iv) s 2 d ≤ o ( m ) .
There exist measures D + and D − over [ m ] ∪ { 0 } and a union J of d points in [ m ] ∪ { 0 } such that: 1.
(a) D + = 0 on J , and (b) D + > 2 D − on J =
[ m ] ∪ { 0 } \ J . 2.
All but ζ -fraction of the measure of D − lies in J . 3.
The distributions
D + / (cid:107)D
+ (cid:107) 1 and D − / (cid:107)D − (cid:107) 1 satisfy Condition 3.3 with parameters k and ν ≤ (cid:0) m k (cid:1) exp( − Ω( m/s 2 )) .
(a) D
+ is at most O (1)Bin( m, 1 / 2) and (b) (cid:107)D + (cid:107) 1 = Θ(1) .
(cid:107) D − (cid:107) 1 = Θ(1 /s ) .
Proof.
We start by constructing each measure in turn.
Deﬁnition of the Measure D − .
We deﬁne the measure D − as follows: D − ( x ) :=
Bin( m, 1 / 2)( x ) if x ≡ 0 (mod s ) ; otherwise D − ( x ) = 0 .
We claim that this satisﬁes Condition 3.3.
This is shown in the following lemma.
Lemma 3.7.
D − ( x ) satisﬁes Condition 3.3 with parameters k
and ν = s (cid:0) mk (cid:1) exp( − Ω( m/s 2 )) .
Proof.
We need to bound E Z ∼D − [ K t ( Z ; m )] for 1 ≤ t ≤ k .
By deﬁnition, we have that
[Equation]
where Y has Z 1’s and m − Z 0’s, and R ∈ { 0 , 1 } m is the unique symmetric measure with (cid:80) mi
=1 X
i having measure D − , and T 0 ⊆
[ m ] is some subset with | T 0 | = t .
Let ω be a primitive s th root of unity.
We note that the pmf R ( x ) of the measure R satisﬁes
[Equation]
Therefore, we can write
[Equation]
Since the expectation is the sum of the above over all x ∈ { 0 , 1 } m and since this separates as a product, we get that
[Equation]
Note that the terms with 2 j ≡ 0 (mod s ) have indices i such that ω j ( − 1)
I [ i ∈ T 0 ]
= − 1 , and do not contribute to the sum.
Other terms will have each value of | 1 + ω j ( − 1)
I [ i ∈ T 0 ] | at most 2 − Ω(1 /s 2 ) .
Therefore, E X ∼R [ χ T 0 ( X )] =
exp( − Ω( m/s 2 )) .
This completes our proof.
We also note that D − is clearly bounded above by Bin( m, 1 / 2) .
We deﬁne J to be the union of the d elements of m ∪ { 0 } congruent to 0 modulo s that are closest to m/ 2 .
We note that the measure of D − outside J is clearly at most the probability that Bin( m, 1 / 2) is more than ds/ 2 from m/ 2 , which is at most ζ by standard tail bounds.
Deﬁnition of the Measure D + .
Intuitively, we would like to deﬁne D + to be equal to some suitable multiple (say, 3 ) of the standard Binomial measure Bin( m, 1 / 2) .
Such a deﬁnition would satisfy the desired moment-matching conditions (property 3 of Proposition 3.6) with zero error and would also guarantee that D + > 2 D − on J , as desired (property 1(b)).
However, this candidate deﬁnition does not satisfy property 1(a), i.e., that D + be equal to 0 on J .
To satisfy the latter property, we will need to carefully modify this measure.
The key lemma is the following (see Appendix B for the proof):
Lemma 3.8.
Let s ≥ ω
( k 4 ) .
There exists a signed measure µ on {− s + 1 , − s + 2 , . . .
, s − 1 } such that:
(i) For any integer 0 ≤ t ≤ k , (cid:80) s − 1 i
=1 − s µ ( i ) i t = 0 , (ii) µ (0) =
− 1 , (iii) | µ ( i ) | < 1 / 10 , i (cid:54) = 0 .
We are now ready to construct the measure D + .
We begin with the measure 3Bin( m, 1 / 2) .
We then for each element x ∈ J take the measure µ from Lemma 3.8, translate it to center around x and add an appropriate multiple of it to D + in order to make D + ( x ) = 0 .
It is clear that the ﬁrst k moments of D + agree with those moments of 3Bin( m, 1 / 2) , and from there it follows that D + satisﬁes
Condition 3.3 with ν = 0 , since for any 0 ≤ t ≤ k and any point x ∈ J , we have that
[Equation]
which means that we never change the moments by making D + ( x ) = 0 .
Therefore, we have D + is 0 on J by our construction.
We also claim that D + is bounded between 2Bin( m, 1 / 2) and 4Bin( m, 1 / 2) on ¯ J .
For this, we note that for any x / ∈ J , there are at most two integers, x (cid:48) and x (cid:48)(cid:48) , that are in J and within distance s of x .
It is clear that
[Equation]
It sufﬁces to show that Bin( m, 1 / 2)( x (cid:48) )
Bin( m, 1 / 2)( x ) < 3 / 2 along with the analogous statement for x (cid:48)(cid:48) .
However, the log of the ratio is easily seen to be O ( s 2 d/m ) =
o (1) , which sufﬁces.
This completes the proof of Proposition 3.6.
3.3 Parameter Setting for the SQ-hard Distributions
We will consider the following family of hardness distributions which will be used in the proof of all SQ hardness results throughout this article.
Let C > 0 be a sufﬁciently large universal constant.
Let m be a positive integer and m (cid:48) be an integer on the order of Cm .
Let d be an integer on the order of m 1 / 10 , s an integer on the order of m 4 / 9 , and k an integer on the order of m 2 / 19 .
Observe that (cid:0) 2 d + m (cid:48) m (cid:48) (cid:1) ≤ ( m (cid:48) )
2 d = exp
( O ( Cm 1 / 10 log( m ))) .
Select m as large as possible so that the above is less than M .
Decreasing M if necessary, we can assume that M = (cid:0) 2 d + m (cid:48) m (cid:48) (cid:1) .
We consider the Veronese mapping V O
( d ) :
R m (cid:48)
→ R M , such that the coordinate functions of V O ( d ) are exactly the monomials in m (cid:48) variables of degree at most O ( d ) .
We deﬁne measures D +
and
D − on [ m ] ∪ { 0 } , as given by Proposition 3.6, with k, s and d as above, and taking log(1 /ζ )
a sufﬁciently small multiple of ( ds ) 2 /m , so that ζ = exp( − Ω( m 4 / 45 )) =
exp( − Ω(log( M ) 8 / 9 )) .
It is easily veriﬁed that these parameters satisfy the assumptions of Proposition 3.6.
For a subset
S ⊆
[ m (cid:48) ] of size m and labels a (cid:54) =
b ∈ R , deﬁne the distribution P D + , D − ,p S,a,b as in Deﬁnition 3.4, with p =
(cid:107)D + (cid:107) 1 / ( (cid:107)D + (cid:107) 1 + (cid:107)D − (cid:107) 1 ) .
We will consider the distribution ( X (cid:48) , Y (cid:48) ) on { 0 , 1 } M × { a, b } by drawing ( X , Y ) from P D + , D − ,p S,a,b and letting X (cid:48)
=
V O ( d ) ( X ) and Y (cid:48) =
Y .
It is easy to see that ﬁnding a hypothesis that predicts y (cid:48) given x (cid:48) is equivalent to ﬁnding a hypothesis for y given x (since y = y (cid:48) and there is a known 1-1 mapping between x and x (cid:48) ).
The pointwise bounds on
D + and D − imply that χ 2 ( D + / (cid:107)D + (cid:107) 1 , Bin( m, 1 / 2))
+ χ 2 ( D − / (cid:107)D − (cid:107) 1 , Bin( m, 1 / 2))
=
O ( s 2 ) .
The parameter ν in Proposition 3.5 is at most sm k exp( − Ω( m/s 2 ))
=
exp( − Ω( m 1 / 9 )) .
Note that as M = exp( (cid:101) O ( m 1 / 10 )) , this is exp( − Ω(log( M ) 1 . 1 )) .
As k is also Ω(log( M ) 1 . 05 ) , we have that τ = exp( − Ω(log( M ) 1 . 05 ))
≤ 1 / poly( M ) .
In the remaining part of this article, we will use x (cid:48) , X (cid:48) , y (cid:48) , Y (cid:48) without clariﬁcation to denote the results of x , X , y, Y after the Veronese mapping
V O ( d ) : R m (cid:48)
→ R M .
Concrete SQ Hardness Results
In this section, we prove our SQ hardness results for Massart learning low-weight half-spaces and
ReLU s. We provide additional SQ hardness results for learning fast convergent activations with respect to L 2 -Massart noise in Appendix D.
4.1 SQ Hardness of Learning Low-Weight Boolean Halfspaces with Massart Noise
In this subsection, we prove the following theorem.
Theorem 4.1 (SQ Hardness for Low-weight Massart Halfspaces on { 0 , 1 } M ) .
Any SQ algorithm that learns {± 1 } -weight halfspaces on { 0 , 1 } M , in the presence of Massart noise with η = 1 / 3 , to 0-1 error better than 1 / poly(log( M )) requires either queries of accuracy better than τ := exp( − Ω(log( M ) 1 . 05 )) or at least 1 /τ statistical queries.
This holds even if the optimal classiﬁer has 0-1 error exp( − Ω(log M ) 8 / 9 ) .
Proof.
Our proof will make use of the SQ framework of Section 3.1 and will crucially rely on the one-dimensional construction of Proposition 3.6.
In this subsection, we ﬁx the labels a = 1 , b = − 1 , and apply the construction in Section 3.3 to obtain the joint distributions ( X , Y ) and ( X (cid:48) , Y (cid:48) ) .
Note that y
=
y (cid:48)
and there is a known 1-1 mapping between x and x (cid:48) , therefore ﬁnding a hypothesis that predicts y (cid:48) given x (cid:48) is equivalent to ﬁnding a hypothesis for y given x .
Claim 4.2.
The distribution ( X (cid:48) , Y (cid:48) ) over { 0 , 1 } M × {± 1 } is a Massart LTF distribution with optimal misclassiﬁcation error OPT Mass ≤ exp( − Ω(log( M ) 8 / 9 )) and Massart noise rate upper bound of η = 1 / 3 .
We defer the proof of the above claim to Appendix C.
We now show that the ( D + , D − , 1 , − 1 , m (cid:48) ) -
Hidden Junta Testing Problem efﬁciently reduces to our learning task.
In more detail, we show that any SQ algorithm that computes a hypothesis h (cid:48) satisfying Pr ( X (cid:48) ,Y (cid:48) )
[ h (cid:48)
( X (cid:48) )
(cid:54) =
Y (cid:48) ] < min( p, 1 − p ) − 2 √ 2 τ can be used as a black-box to distinguish between P D + , D − ,p S,a,b , for some unknown subset S ⊆
[ m ] with | S | = m , and U pm (cid:48) .
Since there is a 1-1 mapping between x ∈ { 0 , 1 } m (cid:48) and x (cid:48) ∈ { 0 , 1 } M , we denote h : { 0 , 1 } m (cid:48)
(cid:55)→ {± 1 } to be h ( x ) = h (cid:48)
( x (cid:48) ) .
We note that we can (with one additional query to estimate the Pr [ h (cid:48)
( X (cid:48) )
(cid:54) =
Y (cid:48) ] within error √ 2 τ ) distinguish between (i) the distribution P D + , D − ,p S,a,b , and (ii) the distribution U pm (cid:48) .
This is because for any h we have that Pr ( X ,Y ) ∼ U pm (cid:48)
[ h ( X ) (cid:54) =
Y ] ≥ min( p, 1 − p ) .
Applying Proposition 3.5, we determine that any SQ algorithm which, given access to a distribution P so that either P = U pm (cid:48) , or P is given by P D + , D − ,p S,a,b for some unknown subset S ⊆
[ m (cid:48) ] with | S | = m , correctly distinguishes between these two cases with probability at least 2 / 3 must either make queries of accuracy better than √ 2 τ or must make at least 2 Ω( m ) τ/ ( χ 2 ( A, Bin( m, 1 / 2))
+ χ 2 ( B, Bin( m, 1 / 2))) statistical queries.
Therefore, it is impossible for an SQ algorithm to learn a hypothesis with error better than min( p, 1 − p ) − 2 √ 2 τ = Θ(1 /s ) − Θ( √ τ )
= 1 / polylog( M ) without either using queries of accuracy better than τ or making at least 2 Ω( m ) τ/ polylog( M ) > 1 /τ many queries.
This completes the proof of the SQ-hardness.
It remains to argue that the underlying halfspaces in the hard instance can be assumed to have {± 1 } weights.
To deal with the weights, we note that g is a degree- 2 d PTF that can be deﬁned as the product of 2 d linear polynomials L i , so that each L i has integer coefﬁcients and the sum of the absolute values of these coefﬁcients is O ( m ) .
This means that g can be deﬁned by a degree- 2 d polynomial with integer coefﬁcients and the sum of whose absolute values is at most O ( m ) 2 d = poly( M ) .
By doubling these coefﬁcients, we can assume that they are all even.
Therefore, the linear threshold function L can be deﬁned by a linear polynomial with even integer weights each of which has absolute value at most W .
If we replace our distribution over { 0 , 1 } M by a distribution over { 0 , 1 } MW by duplicating each coordinate W times (i.e., creating a new distribution with coordinates z i,j for i ∈
[ M ] and j ∈ [ W ] with z i,j = x i for all i, j ), we can rewrite L ( x ) as an LTF L (cid:48)
( z ) , where L (cid:48) has {± 1 } -weights.
This is done by replacing a term a i x i by (cid:80) ( a i + W ) /
2 j = z i,j − (cid:80)
Wj =( a i + W ) / 2+
z
i,j .
This completes the proof of Theorem 4.1.
4.2 SQ Hardness of Learning a Single Neuron with Massart Noise
In this subsection, we prove our SQ hardness result of learning a single neuron with ReLU activation and Massart noise.
The standard ReLU function is deﬁned by ReLU( t ) = max( t, 0) , ∀ t ∈ R .
For technical convenience, we will consider the following linear transformation of the standard ReLU , (cid:92)
ReLU( t ) = − 1 if t < 0 , and (cid:92) ReLU( t )
= − 1 + 2 t otherwise.
We note that our SQ hardness result for the (cid:92) ReLU function applies to the standard ReLU function as well.
Theorem 4.3 (SQ Hardness of Massart Learning ReLUs) .
Any SQ algorithm that learns a single neuron with ReLU activation on R M , in the presence of Massart noise with η = 1 / 3 , within squared error better than 1 / poly(log( M )) requires either queries of accuracy better than τ := exp( − Ω(log( M ) 1 . 05 )) or at least 1 /τ statistical queries.
This holds even if (i) the optimal neuron has squared error exp( − Ω(log M ) 8 / 9 ) , (ii) The X values are supported on { 0 , 1 } M , and (iii) the total weight of the neuron is poly( M ) .
Throughout this subsection, we need the following technical lemma.
Lemma 4.4.
Let J be a union of d points in [ m ] ∪ { 0 } for some odd integer d .
Then there exists a real univariate polynomial p ( x ) of degree O ( d ) such that p ( x ) = 1 , ∀ x ∈ J , and p ( x ) ≤ 0 , ∀ x ∈ ¯ J .
In addition, the absolute value of the coefﬁcients of p ( x ) is at most m O ( d ) =
poly( M ) .
Proof.
Let J = { x 1 , . . .
, x d } .
Deﬁne q ( x ) = − (cid:81) d i =1 ( x − ( x i − 1 / 2))( x − ( x i + 1 / 2)) .
By deﬁnition, we have that q ( x ) > 0 , ∀ x ∈ J , and q ( x ) < 0 , ∀ x ∈ ¯ J .
Then, by polynomial interpolation, there exists a real univariate polynomial r of degree d − 1 such that r ( x i )
=
1 √ q ( x i ) , 1 ≤ i ≤ d .
Consider the real univariate polynomial p ( x ) =
r 2 ( x ) q ( x ) .
For any 1 ≤ i ≤ d , we have that p ( x i ) =
r 2 ( x i ) q
( x i )
= 1 and for any x ∈ ¯ J , we have that p ( x ) ≤ 0 since q ( x ) < 0 , ∀ x ∈ ¯ J .
Finally by polynomial interpolation, we know that the absolute value of every coefﬁcient of r ( x ) , p ( x ) is at most m O ( d )
= poly( M ) .
Proof of Theorem 4.3.
Our proof will make use of the SQ framework of Section 3.1 and will crucially rely on the one-dimensional construction of Proposition 3.6.
In this section, we ﬁx the labels a = − 1 , b = 1 , and apply the construction in Section 3.3 to obtain the joint distributions ( X , Y ) and ( X (cid:48) , Y (cid:48) ) .
Note that y
=
y (cid:48)
and there is a known 1-1 mapping between x and x (cid:48) , therefore ﬁnding a hypothesis that predicts y (cid:48) given x (cid:48) is equivalent to ﬁnding a hypothesis for y given x .
Claim 4.5.
The distribution ( X (cid:48) , Y (cid:48) ) over { 0 , 1 } M × {± 1 } is a Massart single neuron distribution with ReLU activation and with optimal squared error OPT Mass − L2 ≤ exp( − Ω(log( M )
We defer the proof of the above claim to Appendix C.
We now show that the ( D + , D − , − 1 , 1 , m (cid:48) ) -
Hidden Junta Testing Problem efﬁciently reduces to our learning task.
In more detail, we show that any
SQ algorithm that computes a hypothesis
[Equation]
satisfying
[Equation]
can be used as a black-box to distinguish between P D + , D − ,p S,a,b , for some unknown subset S ⊆
[ m (cid:48) ] with | S | = m , and U pm (cid:48) .
Since there is a 1-1 mapping between x ∈ { 0 , 1 } m (cid:48) and x (cid:48) ∈ { 0 , 1 } M , we denote h : { 0 , 1 } m (cid:48)
(cid:55)→ R to be h ( x ) = h (cid:48)
( x (cid:48) ) .
We note that we can (with one additional query to estimate the E [( h (cid:48)
( X (cid:48) )
− Y (cid:48) ) 2 ] within error √ 2 τ ) distinguish between (i) the distribution P D + , D − ,p S,a,b , and (ii) the distribution U pm (cid:48) .
This is because for any h we have that
[Equation]
Applying Proposition 3.5, we determine that any SQ algorithm which, given access to a distribution
P so that either P =
U pm (cid:48) , or P is given by P D
+ , D − ,p S,a,b for some unknown subset S ⊆
[ m (cid:48) ] with | S | = m , correctly distinguishes between these two cases with probability at least 2 / 3 must either make queries of accuracy better than √ 2 τ or must make at least 2 Ω( m ) τ/ ( χ 2 ( A, Bin( m, 1 / 2))
+ χ 2 ( B, Bin( m, 1 / 2))) statistical queries.
Therefore, it is impossible for an SQ algorithm to learn a hypothesis with error better than 4 p − 4 p 2 − 2 √ 2 τ = Θ(1 /s ) − Θ( √ τ )
= 1 / polylog( M ) without either using queries of accuracy better than τ or making at least 2 Ω( m ) τ/ polylog( M ) > 1 /τ many queries.
This completes the proof of Theorem 4.3.
Conclusion
In our work, we prove that no efﬁcient SQ algorithm can approximate the optimal error within any constant factor for learning single neurons with Massart noise.
Our “Massart Noise Model" (Deﬁnition 1.1) is a natural generalization of the classic Massart noise for binary classiﬁcation problem, that has been studied in several prior works (including [DPT21, CKMY21]).
In particular, our result answers an open problem proposed in [DPT21].
From technical perspective, we construct new moment matching distributions corresponding to Boolean halfspaces with Massart noise, which is our main technical contribution.
Importantly, our construction has some additional desired properties which allows us to give hardness for learning low-weight LTFs, which is stronger than the result in [DK20].
In addition, we provide a novel technique for transforming our binary construction into hardness of learning real-valued single neurons with Massart noise.