Cryptographic Hardness of Learning Halfspaces with Massart Noise
Abstract
We study the complexity of PAC learning halfspaces in the presence of Massart noise.
In this problem, we are given i.i.d.
labeled examples ( x , y ) ∈ R N × {± 1 } , where the distribution of x is arbitrary and the label y is a Massart corruption of f ( x ) , for an unknown halfspace f : R N → {± 1 } , with flipping probability η ( x ) ≤ η < 1 / 2 .
The goal of the learner is to compute a hypothesis with small 0-1 error.
Our main result is the first computational hardness result for this learning problem.
Specifically, assuming the (widely believed) subexponential-time hardness of the Learning with Errors (LWE) problem, we show that no polynomialtime Massart halfspace learner can achieve error better than Ω( η ) , even if the optimal 0-1 error is small, namely OPT = 2 − log c ( N ) for any universal constant c ∈ (0 , 1) .
Prior work had provided qualitatively similar evidence of hardness in the Statistical Query model.
Our computational hardness result essentially resolves the polynomial PAC learnability of Massart halfspaces, by showing that known efficient learning algorithms for the problem are nearly best possible.
Introduction
A halfspace or linear threshold function (LTF) is any function h w ,t : R N → {± 1 } of the form h w ,t ( x ) :
= sign( ⟨ w , x ⟩ − t ) , where the vector w ∈ R N is called the weight vector, t ∈ R is called the threshold, and sign : R → {± 1 } is defined by sign( t )
= {+
+} if t ≥ 0 and sign( t ) = −otherwise.
Halfspaces are a central concept class in machine learning, extensively investigated since the 1950s [Ros58, Nov62, MP68].
Here we study the computational complexity of learning halfspaces in Valiant’s (distribution independent)
PAC model [Val84], when the labels have been corrupted by Massart noise [MN06].
We define the Massart noise model below.
Definition 1.1 (Massart Noise) .
We say that a joint distribution D of labeled examples ( x , y ) , supported on R N × {± 1 } , satisfies the Massart noise condition with noise parameter η ∈ [0 , 1 / 2) with respect to a concept class C of Boolean-valued functions on R N if there is a concept c ∈ C such that for all x 0 ∈ R N we have that η ( x 0 ) def =
Pr ( x ,y ) ∼ D [ c ( x ) ̸ = y | x = x 0 ] ≤ η .
The Massart PAC learning problem for the concept class C is the following:
Given i.i.d.
samples from a Massart distribution D , as in Definition 1.1, the goal is to output a hypothesis with small 0error.
In this work, we study the computational complexity of the Massart PAC learning problem, when the underlying concept class C is the class of halfspaces on R N . In its above form, the Massart noise model was defined in [MN06]. An essentially equivalent noise model had been defined in the 80s by Sloan and Rivest [Slo88, RS94, Slo96], and a very similar definition had been considered even earlier by Vapnik [Vap82]. The Massart model is a classical semi-random noise model that is more realistic than Random
Classification Noise (RCN) In contrast to RCN, Massart noise allows for variations in misclassification rates (without a priori knowledge of which inputs are more likely to be misclassified).
Asymmetric misclassification rates arise in a number of applications, including in human annotation noise [BK09]. Consequently, learning algorithms that can tolerate Massart noise are less brittle than those that depend on the uniformity of RCN. The agnostic model [Hau92, KSS94], where the noise can be fully adversarial, is of course even more robust; unfortunately, it is computationally hard to obtain agnostic learners with any non-trivial guarantees, even for basic settings. We now return to the class of halfspaces, which is the focus of this work.
We recall that PAC learning halfspaces with RCN is known to be solvable in polynomial time (to any desired accuracy) [BFKV96]. On the other hand, agnostic PAC learning of halfspaces is known to computationally hard (even for weak learning) [GR06, FGKP06, Dan16].
The computational task of PAC learning halfspaces corrupted by Massart noise is a classical problem in machine learning theory that has been posed by several authors since the 1980s [Slo88, Coh97, Blu03]. Until recently, no progress had been made on the efficient PAC learnability of Massart halfspaces. [DGT19] made the first algorithmic progress on this problem: they gave a poly( N, 1 /ϵ ) -time learning algorithm with error guarantee of η + ϵ .
Subsequent work made a number of refinements to this algorithmic result, including giving an efficient proper learner [CKMY20] and developing an efficient learner with strongly polynomial sample complexity [DKT21].
In a related direction, [DIK + 21] gave an efficient boosting algorithm achieving error η + ϵ for any concept class, assuming the existence of a weak learner for the class.
The error bound of η can be very far from the information-theoretically optimum error of OPT , where
OPT = R LTF
( D ) ≤ η .
Indeed, known polynomial-time algorithms only guarantee error ≈ η even if OPT is very small, i.e., OPT ≪
η . This prompts the following question:
Question 1.1.
Is there an efficient learning algorithm for Massart halfspaces with a relative error guarantee?
Specifically, if OPT ≪ η is it possible to achieve error significantly better than η ?
Our main result (Theorem 1.2) answers this question in the negative, assuming the subexponentialtime hardness of the classical Learning with Errors (LWE) problem (Assumption 2.4).
In other words, we essentially resolve the efficient PAC learnability of Massart halfspaces , under a widely-believed cryptographic assumption.
Before we state our main result, we recall the setup of the Learning with Errors (LWE) problem. In the LWE problem, we are given samples ( x 1 , y 1 ) , . . .
, ( x m , y m ) and the goal is to distinguish between the following two cases: (i) Each x i is drawn uniformly at random (u.a.r.) from Z nq , and there is a hidden secret vector s ∈ Z nq
such that y i = ⟨ x i , s ⟩
+ z i , where z i [-
-]∈
Z q is discrete Gaussian noise (independent of x i ); (ii) Each x i and each y i are independent and are sampled u.a.r.
from Z nq and Z q respectively.
Formal definitions of LWE (Definition 2.3) and related distributions together with the precise computational hardness assumption (Assumption 2.4) we rely on are given in Section 2.
Our main result can now be stated as follows:
Theorem 1.2 (Informal Main Theorem) .
Assume that LWE cannot be solved in 2 n 1 − Ω(1) time.
Then, for any constant ζ > 0 , there is no polynomial-time learning algorithm for Massart halfspaces on R N that can output a hypothesis with 0-1 error smaller than Ω( η ) , even when OPT ≤ 2 − log 1 − ζ N and the Massart noise parameter η is a small positive constant.
The reader is also referred to Theorem D.1 in the Appendix for a more detailed formal statement. Theorem 1.2 is the first computational hardness result for PAC learning halfspaces (and, in fact, any non-trivial concept class) in the presence of Massart noise.
Our result rules out even improper PAC learning, where the learner is allowed to output any polynomially evaluatable hypothesis.
As a corollary, it follows that the algorithm given in [DGT19] is essentially the best possible, even when assuming that OPT is almost inverse polynomially small (in the dimension N ).
We also remark that this latter assumption is also nearly the best possible: if OPT is o ( ϵ/N ) , then we can just draw Ω( N/ϵ ) samples and output any halfspace that agrees with these samples.
We note that a line of work has established qualitatively similar hardness in the Statistical Query (SQ) model [Kea98] — a natural, yet restricted, model of computation. Specifically, [CKMY20] established a super-polynomial SQ lower bound for learning within error of OPT + o (1) . Subse- quently, [DK22] gave a near-optimal super-polynomial SQ lower bound: their result rules out the existence of efficient SQ algorithms that achieve error better than Ω( η ) , even if OPT = 2 log 1 − ζ N .
Building on the techniques of [DK22], more recent work [NT22] established an SQ lower bound for learning to error better than η , even if OPT = 2 log 1 − ζ N — matching the guarantees of known algorithms exactly.
While the SQ model is quite broad, it is also restricted.
That is, the aforementioned prior results do not have any implications for the class of all polynomial-time algorithms.
Interestingly, as we will explain in the proceeding discussion, our computational hardness reduction is inspired by the SQ-hard instances constructed in [DK22].
1.2 Brief Technical Overview
Here we give a high-level overview of our approach.
Our reduction proceeds in two steps.
The first is to reduce the standard LWE problem (as described above) to a different “continuous” LWE problem more suitable for our purposes.
In particular, we consider the problem where the x samples are taken uniformly from R n / Z n , y is either taken to be an independent random element of R / Z or is taken to be ⟨ x , s ⟩ mod 1 plus a small amount of (continuous) Gaussian noise, where s is some unknown vector in {± 1 } n .
This reduction follows from existing techniques [Mic18a, GVV22].
The second step — which is the main technical contribution of our work — is reducing this continuous LWE problem to that of learning halfspaces with Massart noise.
The basic idea is to perform a rejection sampling procedure that allows us to take LWE samples ( x , y ) and produce some new samples (˜ x , ˜ y ) .
We will do this so that if y is independent of x , then ˜ y is (nearly) independent of ˜ x ; but if y = ⟨ x , s ⟩ + noise, then ˜ y is a halfspace of ˜ x with a small amount of Massart noise.
An algorithm capable of learning halfspaces with Massart noise (with appropriate parameters) would be able to distinguish these cases by learning a hypothesis h and then looking at the probability that
[Equation]
In the case where ˜ y was a halfspace with noise, this would necessarily be small; but in the case where ˜ x and ˜ y were independent, it could not be.
In order to manage this reduction, we will attempt to produce a distribution (˜ x , ˜ y ) similar to the
SQ-hard instances of Massart halfspaces constructed in [DK22].
These instances can best be thought of as instances of a random variable ( x ′ , y ′ ) in R n × {± 1 } , where y ′ is given by a low-degree polynomial threshold function (PTF) of x ′ with a small amount of Massart noise.
Then, letting ˜ x be the Veronese map applied to x ′ , we see that any low-degree polynomial in x ′ is a linear function of ˜ x , and so ˜ y
=
y ′ is an LTF of ˜ x plus a small amount of Massart noise.
As for how the distribution over ( x ′ , y ′ ) is constructed in [DK22], essentially the conditional distribution of x ′ on y ′ = 1 and on y [-
-]′ =
− 1 are carefully chosen mixtures of discrete Gaussians in the v -direction (for some randomly chosen unit vector v ), and independent standard Gaussians in the orthogonal directions.
() Our goal will be to find a way to perform rejection sampling on the distribution ( x , y ) to produce a distribution of this form.
In pursuit of this, for some small real number b and some a ∈ [0 , b ) , we let x ′ be a random Gaussian subject to x ′ ≡ b x (mod b ) (in the coordinate-wise sense) conditioned on by ≡ a (mod b ) .
We note that if we ignore the noise in the definition of y , this implies that ⟨ x ′ , s ⟩ ≡ ⟨ b x , s ⟩ ≡ b ⟨ x , s ⟩ ≡ by ≡ a (mod b ) (recalling that s ∈ {± 1 } n ).
In fact, it is not hard to see that the resulting distribution on x ′ is close to a standard Gaussian conditioned on ⟨ x ′ , s ⟩ ≡ a (mod b ) .
In other words, x ′ is close to a discrete Gaussian with spacing b/ ∥ s ∥ 2 and offset a/ ∥ s ∥ 2 in the s -direction, and an independent standard Gaussian in orthogonal directions.
Furthermore, this x ′ can be obtained from ( x , y ) samples by rejection sampling: taking many samples until one is found with by ≡ a (mod b ) , and then returning a random x ′ with x ′ ≡ b x (mod b ) .
By taking an appropriate mixture of these distributions, we can manufacture a distribution close to the hard instances in [DK22].
This intuition is explained in detail in Section 3.1; see Lemma 3.3.
(We note that Lemma 3.3 is included only for the purposes of intuition; it is a simpler version of Lemma 3.5, which is one of the main lemmas used to prove our main theorem.)
Unfortunately, as will be discussed in Section 3.2, applying this construction directly does not quite work.
This is because the small noise in the definition of y leads to a small amount of noise in the final values of ⟨ x ′ , s ⟩ .
This gives us distributions that are fairly similar to the hard instances of [DK22], but leads to small regions of values for u , where the following condition holds: Pr ( y [-
-]′ =
+
| x ′ = u ) =
Pr ( y [-
-]′ =
− 1 | x ′ = u ) . Unfortunately, the latter condition cannot hold if y ′ is a function of x ′ with Massart noise.
In order to fix this issue, we need to modify the construction by carving intervals out of the support of x ′ conditioned on y [-
-]′ =
− 1 , in order to eliminate these mixed regions.
This procedure is discussed in detail in Section 3.3.2. 1.3 Additional Related Work
There have also been several recent works showing reductions from LWE or lattice problems to other learning problems.
Concurrent and independent work to ours [Tie22] showed hardness of weakly agnostically learning halfspaces, based on a worst-case lattice problem (via a reduction from “continuous” LWE). Two recent works obtained hardness for the unsupervised problem of learning mixtures of Gaussians (GMMs), assuming hardness of (variants of) the LWE problem. Specifically, [BRST21] defined a continuous version of LWE (whose hardness they established) and reduced it to the problem of learning GMMs. More recently, [GVV22] obtained a direct reduction from LWE to a (different) continuous version of LWE; and leveraged this connection to obtain quantitatively stronger hardness for learning GMMs.
It is worth noting that for the purposes of our reduction, we require as a starting point a continuous version of LWE that differs from the one defined in [BRST21]. Specifically, we require that the distribution on x is uniform on [0 , 1] n (instead of a Gaussian, as in [BRST21]) and the secret vector is binary. The hardness of this continuous version essentially follows from [Mic18b, GVV22].
For x , s ∈ R n with s ̸ = 0 , let x s def = ⟨ x , s ⟩ / ∥ s ∥ 2 be the length of the projection of x in the s direction, and x ⊥ s ∈ R n − 1 be the projection 1 of x on the orthogonal complement of s .
For f, g : U → R , we write f ( u ) ∝ g ( u ) if there is c ∈ R such that f ( u ) = cg ( u ) for all u ∈ U .
We use X ∼ D to denote a random variable X with distribution D .
We use P D or P X for the corresponding probability mass function (pmf) or density function (pdf), and Pr D or Pr X for the measure function of the distribution.
We use D X to denote the distribution of the random variable X .
For S ⊆ R n , we will use λ ( S ) to denote the n -dimensional volume of S .
Let U ( S ) denote the uniform distribution on S .
For a distribution D on R n and S ⊆ R n , we denote by D | S the conditional distribution of
X ∼ D given X ∈ S .
Let D s (resp.
D ⊥ s ) be the distribution of x s (resp.
x ⊥ s ), where x ∼ D .
For distributions D 1 , D 2 , we use D 1 + D 2 to denote the pseudo-distribution with measure function
Pr D 1 + D 2 ( A ) =
Pr D 1 ( A )
+ Pr D 2 ( A ) .
For a ∈ R , let a D denote the pseudo-distribution with measure function a Pr D .
On the other hand, let a ◦ D denote the distribution of aX , where X ∼ D .
We use D 1 ⋆ D 2 to denote the convolution of distributions D 1 , D 2 .
We will use LTF N for the class of halfspaces on R N ; when N is clear from the context, we may discard it and simply write LTF .
For q ∈ N , we use Z q def = { 0 , 1 , · · · , q − 1 } and R q def = [0 , q ) .
We use mod q : R n (cid:55)→ R nq to denote the function that applies mod q ( x ) on each coordinate of x .
We use D N R n ,σ to denote the n -dimensional Gaussian distribution with mean 0 and covariance matrix σ 2 / (2 π ) · I n and use D N σ as a short hand for D N R ,σ . In some cases, we will use N ( 0 , I n ) for the standard (i.e., zero mean and identity covariance) multivariate Gaussian,
Definition 2.1 (Partially Supported Gaussian Distribution) .
For σ ∈ R + and x ∈ R n , let ρ σ ( x ) def = σ − n exp (cid:0) − π ( ∥ x ∥ 2 /σ ) 2 (cid:1) .
For any countable set S ⊆ R n , we let ρ σ ( S ) def = (cid:80) x ∈ S ρ σ ( x ) , and let
D N S,σ be the distribution supported on S with pmf P D N S,σ ( x ) = ρ σ ( x ) /ρ σ ( S ) .
Definition 2.2 (Discrete Gaussian) . For T ∈ R + , y ∈ R and σ ∈ R + , we define the “ T -spaced, y -offset discrete Gaussian distribution with σ scale” to be the distribution of D N T Z + y,σ .
Learning with Errors (LWE) We use the following definition of LWE, which allows for flexible distributions of samples, secrets, and noises.
Here m is the number of samples, n is the dimension, and q is the ring size.
Definition 2.3 (Generic LWE) .
Let m, n, q ∈ N , and let D sample , D secret , D noise be distributions on R n , R n , R respectively.
In the LWE( m, D sample , D secret , D noise , mod q ) problem, we are given m independent samples ( x , y ) and want to distinguish between the following two cases: (i) Alternative hypothesis : s is drawn from D secret .
Then, each sample is generated by taking x ∼ D sample , z ∼
D noise , and letting y = mod q ( ⟨ x , s ⟩ + z ) ; and (ii) Null hypothesis : x , y are independent and each has the same marginal distribution as above.
When a distribution in LWE is uniform over some set S , we may abbreviate U ( S ) merely as S .
Note that LWE( m, Z nq , Z nq , D N Z ,σ , mod q ) to the classical LWE problem.
Computational Hardness Assumption for LWE As alluded to earlier, the assumption for our hardness result is the hardness of the (classic) LWE problem, with the parameters stated below.
Assumption 2.4 (Standard LWE Assumption (see, e.g., [LP11])) .
Let c > 0 be a sufficiently large constant.
For any constant β ∈ (0 , 1) , κ ∈ N , LWE(2 O ( n β ) , Z nq , Z nq , D N Z ,σ , mod q ) with q ≤ n κ and σ = c √ n cannot be solved in 2 O ( n β ) time with 2 − O ( n β ) advantage. We recall that [Reg09, Pei09] gave a polynomial-time quantum reduction from approximating (the decision version of) the Shortest Vector Problem (GapSVP) to LWE (with similar n, q, σ parameters).
Our hardness assumption is the widely believed sub-exponential hardness of LWE.
We note that the fastest known algorithm for GapSVP takes 2 O ( n ) time [ALNS20].
Thus, refuting the conjecture would be a major breakthrough.
A similar assumption was also used in [GVV22] to establish computational hardness of learning Gaussian mixtures.
Our use of a sub-exponential hardness of LWE is not a coincidence; see Section 4.
As mentioned earlier, we will use a different variant of LWE, where the sample is from R n 1 , the secret is from {± 1 } n , and the noise is drawn from a continuous Gaussian distribution.
The hardness of this variant is stated below. The proof, which follows from [Mic18a, GVV22], is deferred to Appendix B.
Lemma 2.5.
Under Assumption 2.4, for any β ∈ (0 , 1) and γ ∈ R + , there is no 2 O ( n β ) time algorithm to solve LWE (cid:16) 2 O ( n β ) , R n 1 , {± 1 } n , D N O ( n − γ ) , mod 1 (cid:17) with 2 − O ( n β ) advantage.
Decisional Massart Halfspace Problem For a distribution D on labeled examples and a concept class C , we let R C ( D ) def = min h ∈ C Pr ( x ,y ) ∼ D [ h ( x ) ̸ = y ] be the error of the best classifier in C with respect to D .
We will prove hardness for the following decision version of learning Massart halfspaces.
This will directly imply hardness for the corresponding learning (search) problem.
Definition 2.6 (Testing Halfspaces with Massart Noise) .
For n, N ∈ N , η, OPT ∈ (0 , 1 / 2) , let
Massart( m, N, η, OPT) denote the problem of distinguishing, given m i.i.d.
samples from D on R N × {± 1 } , between the following two cases: (i) Alternative hypothesis : D satisfies the Massart halfspace condition with noise parameter η and R LTF ( D ) ≤ OPT ; and (ii) Null hypothesis : the Bayes optimal classifier has cη error, where c > 0 is a sufficiently small universal constant.
Reduction from LWE to Learning Massart Halfspaces
In this section, we establish Theorem 1.2.
Some intermediate technical lemmas have been deferred to the Appendix C. Our starting point is the problem LWE( m, R n 1 , {± 1 } n , D N σ , mod 1 ) .
Note that, by Lemma 2.5, Assumption 2.4 implies the hardness of LWE( m, R n 1 , {± 1 } n , D N σ , mod 1 ) .
We will reduce this variant of LWE to the decision/testing version of Massart halfspaces (Definition 2.6).
Our reduction will employ multiple underlying parameters, which are required to satisfy a set of conditions. For convenience, we list these conditions below.
Condition 3.1.
Let n, m, m ′ ∈ N , t, ϵ, σ ∈ R + , δ ∈ (0 , 1) , satisfy: (i) t/ϵ is a sufficiently large even integer, (ii) σ ≤ √ n , (iii) 1 t √ n ≥ (cid:112) c log( n/δ ) , where c is a sufficiently large universal constant, (iv) ( c ′ ϵ c ′′ tσ ) 2 ≥ log( m ′ /δ ) , where c ′ > 0 is a sufficiently small universal constant and c ′′ > 0 is a sufficiently large universal constant.
The main theorem of this work is stated below.
Theorem 3.2.
Let n, m, m ′ ∈ N , t, ϵ, σ ∈ R + , ϵ ′ , δ ∈ (0 , 1) satisfy Condition 3.1 and η < 1 / 2 .
Moreover, assume that m [-
-]′ =
c ( ϵ/t ) m , where c > 0 is a sufficiently small universal constant and m ( ϵ/t ) 2 is sufficiently large, and N = ( n + 1) d , where d/ ( t/ϵ ) is sufficiently large.
Suppose that there is no T + poly( m, N, log(1 /δ )) -time algorithm for solving LWE( m, R n 1 , {± 1 } n , D N σ , mod 1 ) with ϵ ′ − O ( δ ) advantage.
Then there is no T time algorithm for solving Massart( m ′ , N, η, OPT) with 2 ϵ ′ advantage, where OPT = exp( − Ω( t 4 /ϵ 2 )) .
Note that Theorem 3.2, combined with Lemma 2.5, can be easily used to prove Theorem 1.2 (e.g., by plugging in t = n − 0 .
5 − Θ( ζ ) , ϵ = Θ( n − 1 . 5 ) in the above statement); see Appendix D.
As such, we devote the remainder of the body of this paper to give an overview to the proof of Theorem 3.2.
High-level Overview The starting point of our computational hardness reduction is the family of SQ-hard instances obtained in [DK22].
At a high-level, these instances are constructed using mixtures of “hidden direction” discrete Gaussian distributions, i.e., distributions that are discrete Gaussians in a hidden direction and continuous Gaussians on the orthogonal directions.
In Section 3.1, we note that by using an appropriate rejection sampling procedure on the LWE samples (drawn from the alternative hypothesis), we obtain a distribution very similar to the “hidden direction discrete Gaussian”.
A crucial difference in our setting is the existence of a small amount of additional “noise”.
A natural attempt is to replace the discrete Gaussians in [DK22] with the noisy ones obtained from our rejection sampling procedure.
This produces problems similar to the hard instances from [DK22].
Unfortunately, the extra noise in our construction means that the naive version of this construction will not work; even with small amounts of noise, the resulting distributions will not satisfy the assumptions of a PTF with Massart noise.
In Section 3.2, we elaborate on this issue and the modifications we need to make to our construction in order to overcome it.
In Section 3.3, we provide the complete construction of our Massart PTF hard instance.
Overview of the [DK22] SQ-hard Construction [DK22] showed SQ-hardness for the following hypothesis testing version of the problem (which implies hardness for the learning problem):
For an input distribution D on R n × {± 1 } , distinguish between the cases where D is a specific distribution D null in which x and y are independent or where D belongs to a class of alternative hypothesis distributions D alternative .
In particular, for D ∈ D alternative , y will be given by a low-degree PTF in x with a small amount of Massart noise.
As we will be trying to reproduce it, it is important for us to understand this alternative hypothesis distribution.
Each distribution in D alternative is parameterized by a hidden direction s ∈ S n − 1 .
We will denote the corresponding distribution by D s .
D s is constructed so that x ⊥ s ∼ D N R n − 1 , 1 is independent of x s and y .
This means that we can specify D s by describing the simpler distribution of ( x s , y ) ∈ R × {± 1 } .
For ( x s , y ) , we have that y
=
+ with probability 1 − η .
The distributions of x s conditioned on y = ± 1 are defined to be mixtures of discrete Gaussians as follows:
[Equation]
As we noted, both x s | ( y = +1) and x s | ( y = − 1) are mixtures of discrete Gaussians.
Combining this with the fact that x ⊥ s ∼ N ( n, I n − 1 ) , this indicates that x | ( y = +1) and x | ( y = − 1) are mixtures of “hidden direction discrete Gaussians” — with different spacing and offset for their support on the hidden direction.
These conditional distributions were carefully selected to ensure that y is a Massart PTF of x with small error.
To see why this is, notice that the support of x s | ( y = +1) is (cid:83) i ∈
[-i 
-] Z [ it, it + ( i + 1) ϵ ] , while the support of x s | ( y = − 1) is (cid:83) i ∈
Z
[ it + t/ 2 , it + t/ 2 + ( i + 1) ϵ ] ; both supports are unions of intervals.
Consider the implications of this for three different ranges of x s :
For x s ∈ [ − t 2 / (2 ϵ ) , t 2 / (2 ϵ )] , the intervals have lengths in [0 , t/ 2] ; thus, the +1 intervals and the − 1 intervals do not overlap at all.
For x s ∈ [ − t 2 /ϵ, − t 2 / (2 ϵ )) ∪ ( t 2 / (2 ϵ ) , t 2 /ϵ ] , the intervals have lengths in [ t/ 2 , t ] ; thus, the + intervals and the − 1 intervals overlap, so that their union covers the space.
We note that in this case there are gaps between the +1 intervals; specifically, there are at most O ( t/ϵ ) such gaps.
For x s ∈ ( −∞ , − t 2 /ϵ ) ∪ ( t 2 /ϵ, ∞ ) , the intervals have lengths in [ t, ∞ ) , so the +1 intervals cover the space by themselves.
Consider the degree- O ( t/ϵ ) PTF sign( p ( x )) such that sign( p ( x ))
=
[-+
-]+1 iff x s ∈ (cid:83) i [-
-]∈
Z [ it, it +( i +1) ϵ ] .
In particular, sign( p ( x ))
= 1 for x in the support of the conditional distribution on y = 1 .
Note that the PTF sign( p ( x )) has zero error in the first case; thus, its total 0-1 error is at most exp( − Ω( t 2 /ϵ ) 2 ) .
Moreover, since the probability of y = 1 is substantially larger than the probability of y = − 1 , it is not hard to see that for any x with sign( p ( x ))
= {+
+} that Pr [ y = 1 | x = x ] > 1 − O ( η ) .
This implies that y is given by sign( p ( x )) with Massart noise O ( η ) .
3.1 Basic Rejection Sampling Procedure
In this subsection, we show that by performing rejection sampling on LWE samples, one can obtain a distribution similar to the “hidden direction discrete Gaussian”.
For the sake of intuition, we start with the following simple lemma.
The lemma essentially states that, doing rejection sampling on LWE samples, gives a distribution with the following properties: On the hidden direction s , the distribution is pointwise close to the convolutional sum of a discrete Gaussian and a continuous Gaussian noise.
Moreover, on all the other directions ⊥ s , the distribution is nearly independent of its value on s , in the sense that conditioning on any value on s , the distribution on ⊥ s stays pointwise close to a Gaussian.
Note that this distribution closely resembles the “hidden direction discrete Gaussian” in [DK22].
Lemma 3.3.
Let ( x , y ) be a sample of the LWE( m, R n 1 , {± 1 } n , D N σ , mod 1 ) from the alternative hypothesis case, let y ′ be any constant in [0 , 1) , and let x ′ ∼ (1 /σ scale ) ◦ D N x + Z n ,σ scale | ( y = y ′ ) .
Then we have the following: (i) For x ′ s , we have that for any u ∈ R it holds that P x ′ s ( u ) =
(1 ± O ( δ ))
P D ′ ⋆D N σ noise ( u ) , where D ′ =
D N T ( y ′ + Z ) ,σ signal , and T = SR / ( n 1 / 2 σ scale ) , σ signal = √ SR , σ noise = √ 1 − SR , and SR = σ 2scale
σ 2scale + σ 2 /n , (ii) x ′⊥ s is “nearly independent” of x ′ s , namely for any l ∈ R and u ∈ R n − 1 we have that P x ⊥ s | x s = l ( u ) =
(1 ± O ( δ )) P D N R n − 1 , 1 ( u ) .
Lemma 3.3 is a special case of Lemma 3.5, which is one of the main lemmas required for our proof.
We note that the distribution of x ′ obtained from the above rejection sampling is very similar to the “hidden direction discrete Gaussian” used in [DK22].
The key differences are as follows: (i) on the hidden direction, x ′ s is close to a discrete Gaussian plus extra Gaussian noise (instead of simply being a discrete Gaussian), (ii)
x ′⊥ s and x ′ s are not perfectly independent.
More importantly, by taking different values for y ′ and σ scale , we can obtain distributions with the same hidden direction, but their discrete Gaussian on the hidden direction has different spacing ( T ) and offset ( y ′ ).
To obtain a computational hardness reduction, our goal will be to simulate the instances from [DK22] by replacing the hidden direction discrete Gaussians with the noisy versions that we obtain from this rejection sampling.
We next discuss this procedure and see why a naive implementation of it does not produce a PTF with Massart noise.
3.2 Intuition for the Hard Instance
The natural thing to try is to simulate the conditional distributions from [DK22] by replacing the hidden direction discrete Gaussian terms in (1) with similar distributions obtained from rejection sampling.
In particular, Lemma 3.3 says that we can obtain a distribution which is close to this hidden direction Gaussian plus a small amount of Gaussian noise.
Unfortunately, this extra noise will cause problems for our construction.
Recall that the support of x s | ( y = +1) was (cid:83)
i [-
-]∈
Z [ it, it + ( i + 1) ϵ ] , and the support of x s | ( y = − 1) was (cid:83) i ∈
Z [ it + t/ 2 , it + t/ 2 + ( i + 1) ϵ ] for [DK22].
With the extra noise, there is a decaying density tail in both sides of each [ it, it + ( i + 1) ϵ ] interval in the support of x s | ( y = +1) .
The same holds for each interval in the support of x s | ( y = − 1) .
Recalling the three cases of these intervals discussed earlier, this leads to the following issue.
In the second case, the intervals have length within [ t/ 2 , t ] ; thus, the intervals [ it, it + ( i + 1) ϵ ] and [ it + t/ 2 , it + t/ 2 + ( i + 1) ϵ ] overlap, i.e., it + ( i + 1) ϵ ≥ it + t/ 2 .
On the right side of [ it, it + ( i + 1) ϵ ] , in the support of x s | ( y = − 1) , there is a small region of values for u , [-it + t/ 
-]where Pr [ y ′ =
{++
 |+} x s = u ] = Pr [ y ′ = − 1 | x s = u ] .
This causes the labels y
= +
and y = − 1 to be equally likely over that small region, violating the Massart condition.
(We note that for the first case, there is also this kind of small region that Pr [ [-it + t/ 2 , it + t/ 
-]{+y ′ = +
+}
| x s = u ] =
Pr [ y ′ =
− 1 | x s = u ] caused by the noise tail.
However, the probability density of this region is negligibly small, as we will later see in Lemma 3.9.)
{+We can address this by carving out empty spaces in the [ it + t/ 2 , it + t/ 
+}
+ ( i + 1) ϵ ] intervals for x s | ( y = − 1) , so that these decaying parts can fit into.
Since this only needs to be done for intervals of Case 2, at most O ( t/ϵ ) many such slots are needed.
It should be noted that no finite slot will totally prevent this from occurring. However, we only need the slot to be wide enough so that the decay of the error implies that there is negligible mass in the overlap (which can be treated as an error).
We also need to discuss another technical detail.
In the last section, we defined the rejection sampling process as taking (1 /σ scale ) ◦ D N x + Z n ,σ scale | ( y = y ′ ) , where we can control the offset by y ′ and spacing by σ scale (Lemma 3.3).
This distribution is effectively a noisy version of a discrete Gaussian. Therefore, we can produce a noisy version of the hard instances of [DK22] by taking a mixture of these noisy discrete Gaussians. Unfortunately the noise rate of one of these instances will be σ noise .
This quantity depends on the spacing T of the discrete Gaussian, which varies across the mixture we would like to take. This inconsistent noise rate is inconvenient for our analysis. However, we can fix the issue by adding extra noise artificially to each of the discrete Gaussians in our mixture, so that they will all have a uniform noise rate σ noise ; see Algorithm 1 and Lemma 3.5.
The last bit of technical detail is that instead of doing the rejection for y = y ′ , which has 0 acceptance probability, we will only reject if y is not corresponding to any discrete Gaussian we need.
Then we do another rejection to make sure that the magnitude of discrete Gaussians in the mixture is correct.
In the next subsection, we introduce the complete rejection sampling method.
3.3 The Full Hard Instance Construction
We first introduce the complete rejection algorithm, and then explain how the hard instance is produced using it.
Below we provide proof overviews; omitted proofs can be found in Appendix C.
3.3.1 The Complete Rejection Algorithm
The rejection sampling algorithm is the following.
The sampling process produces the noisy variant of the distribution which, for some carefully selected set B ⊆ [0 , 1] , has PDF functionλ ( B ) (cid:82) B D N k
+( t + k − ψ ) Z , 1 dk in the hidden direction, as we will see in Lemma 3.5.
Inputs: A sample ( x , y ) ∈ R n 1 × R 1 and the input parameters are t, ϵ, ψ ∈ R > 0 , where ψ + ϵ ≤ t , B ⊆ [ ψ, ψ + ϵ ] , δ ∈ (0 , 1) .
In addition, the parameters satisfy items (i)-(iii) of Condition 3.1. Output:
REJECT or a sample x ′ ∈ R n . Reject unless there is a k ∈ B
such that y =
k t + k − ψ . 2.
Furthermore, reject with probability 1 − t 2 ( t + k − ψ ) 2 . 3.
Let SR = 1 − 4( t + ϵ ) 2 σ 2 , σ scale =
SR ( t + k − ψ ) √ n and σ add = (cid:113) (1 − SR) σ 2scale − SR( σ/ √ n ) 2 SR .
Then, sample independent noise x add ∼ D N R n ,σ add and output x ′ ∼ (1 /σ scale ) ◦ D N x + x add + Z n ,σ scale .
Notice that the parameter SR does not depend on y , whereas σ scale , σ add do depend on y .
For convenience, let us use the following notation for the output distributions.
Definition 3.4 (Output Distribution of Rejection Sampling) .
Let D alternative t,ϵ,ψ,B,δ be the distributions of x ′ produced by Algorithm 1 (conditioned that the algorithm accepts) given that ( x , y ) are sampled as follows: let x ∼ U ( R n 1 ) , z ∼ D N σ , and then let y
= mod 1 ( ⟨ x , s ⟩ + z ) , where s ∈ {± 1 } n is the secret.
Furthermore, let D null t,ϵ,ψ,B,δ be a similar distribution, but when x ∼ U ( R n 1 ) , y ∼ U ( R 1 ) are independent.
Note that D alternative t,ϵ,ψ,B,δ depends on s , but we do not explicitly denote this in our notation.
Alternative Hypothesis Analysis The main properties of D alternative t,ϵ,ψ,B,δ are summarized in the following lemma.
Essentially, the lemma states that for this distribution D alternative t,ϵ,ψ,B,δ , the marginal distribution on the hidden direction s is pointwise close to the convolution sum of D ′ and a Gaussian noise, where D ′ is a linear combination of discrete Gaussians.
Moreover, on all the other directions ⊥ s , the distribution is nearly independent of its value on s , in the sense that conditioning on any value on s , the distribution on ⊥ s always stays pointwise close to a Gaussian.
Lemma 3.5.
Let x ′ ∼ D alternative t,ϵ,ψ,B,δ .
Then we have the following: (i) For x ′ s , we have that for any u ∈ R ,
P x ′ s ( u ) = (1 ± O ( δ ))
P D
[Equation]
⋆D N σ noise ( u ) , where D ′ =
[Equation]
σ signal = √ SR , and σ noise = √ 1 − SR = 2( t + ϵ ) σ .
( SR is defined in Algorithm 1), (ii) x ′⊥ s is “nearly independent” of x ′ s ; namely, for any l ∈ R and u ∈ R n − 1 , we have that P x ′⊥ s | x ′ s
= l ( u ) =
(1 ± O ( δ ))
P D N R n − 1 , 1 ( u )[-
 +(  2 + k s k 22  2add ) = p SR , where theequality follows from k s k 22 = n and our choice of  add-] .
Null Hypothesis Analysis For D null t,ϵ,ψ,B,δ , we can show that it is pointwise close to D N R n , 1 :
Lemma 3.6.
For any u ∈ R n , we have that P D null t,ϵ,ψ,B,δ ( u ) =
(1 ± O ( δ ))
P D N R n, 1 ( u ) .
3.3.2 The Reduction Algorithm
With the rejection sampling algorithm (Algorithm 1) at our disposal, we can now give the full construction of the hard instance.
We use D t,ϵ,ψ + ,B + ,δ for x | y
= +1 , D t,ϵ,ψ − ,B − ,δ for x | y =
− 1 (with a carefully chosen pair of ( B + , ψ + ) and ( B − , ψ − ) , as we discussed in Section 3.2), and take a proper marginal distribution of y to build a joint distribution of ( x , y ) .
We introduce a reduction algorithm that, given samples from our LWE problem (either from the null or the alternative hypothesis), produces i.i.d.
samples ( x , y ) from a joint distribution with the following properties:
If the input LWE problem is the null hypothesis, then x | y
= +1 and x | y
= − 1 are close in total variation distance.
Therefore, no hypothesis for predicting y in terms of x can do much better than the best constant hypothesis.
If the input LWE problem is the alternative hypothesis, then the joint distribution of ( x , y ) we build is close to a distribution D that satisfies O ( η ) Massart condition with respect to a degree- O ( t/ϵ ) PTF, and there is a degree- O ( t/ϵ ) PTF with small error on D .
We formalize the idea from Section 3.2 here.
For x | y
= +1 , we will use ψ + def = 0 and B + def
=
[0 , ϵ ] . For x | y = − 1 , we take ψ − def = t/ 2 , which is also the same as [DK22]; but instead of taking
B − def =
[ t/ 2 , t/ 2 + ϵ ] , we will need to carve out the slots on B − .
First, we define the mapping g : R − [ − 1 . 5 t, 0 .
5 t ] (cid:55)→ [0 . 5 t, t ] , as follows: for i ∈ Z and b ∈ R t , we have that
[Equation] This function maps a location it + t/ 2 + b to the corresponding place we need to carve out on B − , which is defined in Algorithm 2.
These intervals are chosen so that the decaying density part of + can fit in, as we discussed in Section 3.2.
Now we introduce the algorithm that reduces LWE to learning Massart PTFs.
We similarly define the output distributions of the algorithms in the two cases as follows:
Definition 3.7.
Let D alternativePTF be mixture of D alternative t,ϵ,ψ + ,B + ,δ and D alternative t,ϵ,ψ − ,B − ,δ with +1 and − 1 labels and weights 1 − η and η respectively.
Similarly, let D nullPTF be mixture of D null t,ϵ,ψ + ,B + ,δ and D null t,ϵ,ψ − ,B − ,δ with +1 and − 1 labels and weights 1 − η and η respectively.
The following observation is immediate from the algorithm.
Observation 3.8.
In the alternative (resp. null) hypothesis case, the output distribution of Algorithm 2, conditioned on not failing, is the same as m ′ i.i.d.
samples drawn from D alternativePTF (resp.
D nullPTF ).
Alternative Hypothesis Analysis We prove that there exists a degree- O ( t/ϵ ) PTF such that
D alternativePTF is close to (in total variation distance) satisfying the O ( η ) Massart noise condition with respect to this PTF, and this PTF has small error with respect to D alternativePTF .
Lemma 3.9.
D alternativePTF is O ( δ/m ′ ) close in total variation distance to a distribution D truncated such that there is a degree- O ( t/ϵ ) PTF sign( p ( x )) that: (i) E ( x ,y ) ∼ D truncated [sign( p ( x ))
̸
= y ] ≤ exp( − Ω( t 4 /ϵ 2 )) , (ii) D truncated satisfies the O ( η ) Massart noise condition with respect to sign( p ( x )) . Null Hypothesis Analysis The reader is referred to Lemma C.8 in Appendix C for the null hypothesis analysis.
Inputs: m samples from an instance of LWE( m, R n 1 , {± 1 } n , N σ , mod 1 ) .
The input parameters are m ′ ∈ N , t, ϵ ∈ R > 0 , δ ∈ (0 , 1) , and η > 0 a sufficiently small value.
In addition, the parameters satisfy Condition 3.1.
Output: m ′ many samples in R n × {± 1 } or FAIL.
We take ψ + = 0 , B + = [0 , ϵ ] , ψ − = t/ 2 and B − def = [ t/ 2 , t/ 2 + ϵ ] −
[Equation]
Repeat the following m ′ times.
If at any point the algorithm attempts to use more than m LWE samples from the input, then output FAIL.
(a) With probability 1 − η , repeat the following until Algorithm 1 accepts and output x ′ : run Algorithm 1 with the next unused LWE sample from the input and parameters t, ϵ, ψ
= ψ + , B = B + , δ .
Add ( x ′ , +1) to the output samples.
(b) With probability η , repeat the following until Algorithm 1 accepts and output x ′ : run Algorithm 1 with the next unused LWE sample from the input and parameters t, ϵ, ψ = ψ − , B = B − , δ .
Add ( x [-)) 
 = y ]  exp(  ⌦ ( t 4 / ✏ 2 )) + O (  /m 0 ) ; and-]′ , − 1) to the output samples.
Putting Everything Together Having reduced LWE to learning Massart PTFs, we can apply a
Veronese mapping on the samples; this PTF becomes an LTF on the Veronese mapping.
Since we use degree- O ( t/ϵ ) Veronese mapping, the dimension for the Massart LTF problem is N = ( n + 1) O ( t/ϵ ) .
Discussion
Our result rules out the existence of polynomial time algorithms achieving error smaller than Ω( η ) , where η is the upper bound on the noise rate, even of the optimal error is very small, assuming the subexponential time hardness of LWE.
A technical open question is whether the constant factor in the Ω( η ) -term of our lower bound can be improved to the value C = 1 ; this would match known algorithms exactly. (As mentioned in the introduction, such a sharp lower bound has been recently established in the SQ model [NT22], improving on [DK22].) It is also worth noting that our reduction rules out polynomial-time algorithms, but does not rule out, e.g., subexponential or even quasipolynomial time algorithms with improved error guarantees. We believe that obtaining stronger hardness for these problems would require substantially new ideas, as our runtime lower bounds are essentially the same as the best time lower bounds for learning in the (much stronger) agnostic noise model or in restricted models of computation (like SQ).
This seems related to the requirement that our bounds require subexponential hardness of LWE in our assumption.
As the strongest possible assumptions only allow us to prove quasi-polynomial lower bounds, any substantially weaker assumption will likely fail to prove super-polynomial ones. Acknowledgments Ilias Diakonikolas was supported by NSF Medium Award CCF-2107079, NSF Award CCF- (CAREER), a Sloan Research Fellowship, and a DARPA Learning with Less Labels (LwLL) grant. Daniel M. Kane was supported by NSF Medium Award CCF-2107547, NSF Award CCF- (CAREER), a Sloan Research Fellowship, and a grant from CasperLabs.
Lisheng Ren was supported by NSF Award CCF-1652862 (CAREER) and a DARPA Learning with Less Labels (LwLL) grant.