Large (robust) models from computational constraints Anonymous Author(s)
AfﬁliationAddress email
Abstract
Large models with thousands of parameters have been hugely successful.
Inthis work, we ask: can the need for large models be due to the computationallimitation of the learner?
Additionally, we ask, is this situation exacerbated forrobust learning?
We show that this indeed could be the case.
We show learningtasks for which computationally bounded learners need signiﬁcantly more modelparameters than those of information-theoretic learners.
Furthermore, we showthat even more model parameters could be necessary for the case of robust learning.
In particular, for computationally bounded learners, we boost the recent result of Bubeck and Sellke [NeurIPS’2021], which shows that robust models might needmore parameters, to the computational regime and show that bounded learnerscould provably need an even larger number of parameters.
Next, we ask: canwe hope to remedy the situation for robust computationally bounded learningby restricting adversaries to also be computationally bounded?
Here again, weshow that this might be possible.
Speciﬁcally, building on Garg, Jha, Mahloujifar,and Mahmoody [ALT’2020], we demonstrate a learning task that can be learnedrobustly and efﬁciently against a computationally bounded attacker with a smallmodel. On the other hand, to be robust against an information-theoretic attackerrequires the learner to output a much larger model.
Introduction
The fundamental theory of learning theory characterizes the set of hypothesis functions that arelearnable (e g. to have ﬁnite VC dimension) and further shows that the simple “empirical riskminimization” can always do the job.
In particular, in the above (realizable) setting, it is enough tooutput any f 2 H that is consistent with the training set.
This means that if H is a “simple” class, 23 then at least information theoretically , there is no need to have a more complex model than what thehypothesis class H requires.
More speciﬁcally, if all h 2 H can be represented with ` number of 25 bits (also referred to as parameters), then the learner, (again information-theoretically) does not needmore parameters in the output model f than what h needs.
What requires large models?
In recent years, deep neural nets with millions or even billions [-of parameters 
-]ofparameters 1 [Wortsman et al., 2022, Dai et al., 2021, Yu et al., 2022] have come out as one of themost powerful models for very basic tasks such as image classiﬁcation.
A magic of DNNs is thatthey generalize without falling into the classical theories mentioned above, and hence they are thesubject of an active line of work aiming to understand how DNNs generalize Arora et al.
Allen-Zhu et al.
[2019b,a], Novak et al.
[2018], Neyshabur et al.
[2014] and the various beneﬁts ofoverparametrized regimes Xu et al.
[2018], Chang et al.
In fact, the number of parameters insuch models is so large that it is enough to memorize (and ﬁt) to a large number of random labels [Zhang et al., 2021].
This leads us to our ﬁrst main question, in which we study one potential causefor having large models:
Are there any learning tasks for which computationally bounded learners need tooutput signiﬁcantly more model parameters than needed information-theoretically?
One should be cautious in how to formulate the question above.
That is because, many simple(information-theoretically learnable) tasks are believed to be computationally hard to learn (e.g.,learning parity with noise [Pietrzak, 2012]).
In that case, one can interpret this as saying that theefﬁcient learner requires inﬁnite number of parameters, as a way of saying that the learning is notpossible at all!
However, as explained above, we are interested in understanding the reason behindhaving large number of parameters when learning is possible .
Could robustness also be a cause?
A highly sought-after property of machine learning modelsis their robustness to the so-called adversarial examples [Goodfellow et al., 2014].
Here we wouldlike to ﬁnd a model f such that f ( x ) = f
( x 0 ) holds with high probability when x D is anhonestly sampled instance and x 0 ⇡ x is a close instance that is perhaps minimally (yet adversarially)perturbed.
A recent work of Bubeck and Sellke [2021] showed that having large model parameterscould be due to the robustness of the model.
Here, we are asking whether such phenomenon can havea computational variant that perhaps leads to needing even more parameters when the robust learneris running in polynomial time.
Are there any learning tasks for which computationally bounded robust learningcomes at the cost of having even more model parameters than non-robust learning?
Can we remedy the situation?
We ask whether natural restrictions on the adversary can helpreduce model sizes.
In particular, we consider the restriction to the class of polynomially boundedadversaries.
In fact, when it comes to robust learning, the computationally bounded aspect could beimposed both on the learner as well as the adversary [Mahloujifar and Mahmoody, 2019, Bubecket al., 2019, Garg et al., 2020b, Degwekar et al., 2019]. Are there any learning tasks for which computationally bounded robust learning
can be done with fewer model parameters for polynomial-time adversaries incomparison with information-theoretic adversaries?
1.1 Our results
Here we describe our results.
In summary, we prove that under the computational assumption thatone-way functions exist, the answer to both of our main questions above is positive.
Indeed, weshow that the computational efﬁciency of the learner could be the cause of having large models.
In particular, we prove the following theorem, in which a learning task is parameterized by  , ahypothesis class H and a class of input distributions D (see Section 2.1 for formal deﬁnitions).
Theorem 1 (Main results, informal) .
If one-way functions exist, and for parameter  and arbitrarypolynomials n < ↵ <  (e.g., n =  0 .
1 , ↵ =  5 ,  =  10 ) the following holds.
Part 1: There is a learning task parameterized by  and a robustness bound (to limit how much anadversary can perturb the inputs) such that:
• The input/instance size is ⇥ ( n ) .•
There is a robust learner that uses ⇥ (  ) parameters.•
Any polynomial-time learner needs ⇥ ( ↵ ) parameters to learn the task.• Any polynomial-time learner needs ⇥ (  ) parameters to robustly learn the task.
Part 2: There is a learning task parameterized by  and a robustness bound (to limit how much anadversary can perturb the inputs) such that: • The input/instance size is ⇥ ( n ) .•
When the adversary that generates the adversarial examples runs in polynomial time, thereis an (efﬁcient) learner that outputs a model with O (1) parameters that robustly predicts theoutput labels with small error.• Against information-theoretic (computationally unbounded) adversaries that do the instanceperturbations, no learner can produce a model with < ↵ parameters that later robustlymakes the predictions.
In Sections 3 and 4, we explain the high-level ideas behind the proof of Theorem 1 and formalizethese statements.
The full proofs can be found in supplemental material.
Preliminaries
For a distribution X , x X denotes that x is sampled according to X .
We use U S for the uniform distribution over S , and we use U n to represent U { 0 , 1 } n .
For a set S , x S means s U S .
For any two distributions X and Y over a ﬁnite universe ⌦ , the statistical distance between X and Y is deﬁned as
[Equation]
We sometimes write X ⇡ " Y to denote SD ( X, Y ) 6 " .
For a vector x = ( x 1 , . . .
, x n ) 2 F nand a subset S
= { i 1 , . .
, i ` } ✓ { 1 , 2 , . . . , n } , x S denotes ( x i 1 , . . .
, x i ` ) .
For any two vectors x = ( x 1 , . . .
, x n ) and y =
( y 1 , . . .
, y n ) , their Hamming distance is deﬁned as HD ( x, y ) =|
{ i 2 { 1 , 2 , . . . , n } : x i 6 = y i }| .
For a set S and an integer [-
 ⩽-]0 6 t 6 |S| ,  S t  represents the set ofsubsets of S of size t .
I stands for the indicator function.
The base for all logarithms in this paper is 2.
2.1 Deﬁnitions Related to Learning and Attacks
In this section, we present the basic notions and deﬁnitions of what it means to learn a class offunctions.
We use X to denote the inputs and Y = { 0 , 1 } to denote the outputs.
By H we denote a 97 hypothesis class of functions from X to Y .
We use D to denote a distribution over X ⇥ Y .
A learning 98 algorithm, takes a set
S 2 ( X ⇥ Y ) ⇤ and a parameter  and outputs a function f that is supposed 99 to predict a fresh sample from the same distribution that has generated the set S .
The parameter  100 is supposed to capture the complexity of the problem, e.g., by allowing the inputs (and the runningtimes) to grow.
(E.g., this could be the VC dimension, but not necessarily so.)
In particular, weassume that the members of the sets X  , Y  can be represented with poly (  ) bits.
A proper learning 103 for a hypothesis class H outputs f 2 H , while an improper learner is allowed to output arbitrary 104 functions.
By default, we work with improper learning as we do particularly focus on whether thelearning algorithms output a function from the hypothesis class.
For a set S ✓ X and a functionh : X ! Y , by S h we denote the labeled set { ( x, h ( x )) | x 2 S} .
For a distribution D and an 107 oracle-aided algorithm A , by A D we denote giving access to a sampler from D to A .
Deﬁnition 2 (Learning problems and learners) .
We use F = { F  =
( X  , Y  , D  , H  ) }  2 N to 109 denote a family of learning problems where each H  is a set of hypothesis functions mapping X  to
Y  and D  is a set of distributions supported on X  .
• For function " ( · , · ) , we say L " -learns F if
[Equation]
where Risk( h, D  , f ) = Pr x D  [
[Equation]
x ) [Equation] x ) [Equation]
• L outputs models with at most p ( · ) bits of parameters if
[Equation]
• L is an " -robust learner against (all) adversaries of budget r w.r
[Equation]
t. distance metric d if [Equation]
[Equation]
where
Risk ( d,r )
[Equation]
• L runs in polynomial time if there is a polynomial poly ( · ) such that for all  2 N and
S 2 ( X  , Y  ) ⇤ , the running time of L ( S ,  ) is bounded by poly ( |S| ·  )
• L is an " -robust learner against polynomial-time adversaries of budget r w.r.t distance d , if for any family of poly (  ) -time (oracle aided) adversaries A
= { A ( · )  }  2 N we have
[Equation]
where
Risk ( d,r,A
[Equation]
[Equation]
2.2 Cryptographic
Primitives
Deﬁnition 3 (Negligible function) .
A function " (  ) is said to be negligible (simply denoted by" (  )
=
negl (  ) ) if for all polynomial poly (  ) , it holds that " (  ) < 1 / poly (  ) for all sufﬁcientlylarge  .
Deﬁnition 4 (Pseudorandom generator) .
Suppose ↵ >  .
A function f : { 0 , 1 }  ! { 0 , 1 } ↵ is called a pseudorandom generator if for all polynomial-time probabilistic algorithm A , it holds that [Equation] ]
[Equation]
The ratio ↵ /  is refer to as the (multiplicative) stretch of the PRG.
Theorem 5 (Håstad et al.
Pseudorandom generators (with arbitrary polynomial stretch)can be constructed from any one-way functions.
Deﬁnition 6 (Signature scheme) .
A signature scheme consists of three algorithms ( Gen , Sign , Verify ) .
• ( vk , sk ) Gen (1  ) : on input the security parameter 1  , the randomized algorithm Gen
outputs a (public) veriﬁcation key vk and a (private) signing key sk .
•  =
Sign ( sk , m ) : on input the signing key sk and a message m , Sign outputs a signature  .
• b = Verify ( vk , m,  ) : on input a veriﬁcation key vk , a message m , and a signature  , Verify outputs a bit b , and b = 1 indicates tha the signature is accepted.
We require a (secure) signature scheme to satisfy the following properties.
• Correctness.
For every message m , it holds that
[Equation]
• Weak unforgeability.
For any PPT algorithm A and message m , it holds that
[Equation]
and
[Equation]
Theorem 7 (Naor and Yung [1989], Rompel [1990]) .
The signature scheme can be constructed fromany one-way function.
consider a rather weak notion of unforgability.
Here, we require that the adversary cannot forge a signature when he is given only one valid pair of message and signature
This weaker security notion already sufﬁces for our purposes
In the stronger notion, the adversary is allowed to pick m based on the given vk .
2.3 Coding Theory
Let F be a ﬁnite ﬁeld.
A linear code H with block length n and dimension k is a k -dimensionalsubspace of the vector space F n .
The generator matrix G 2 F k ⇥ n maps every message m 2 F kto its encoding, namely, the message m is encoded as m · G .
The distance d of the code H is theminimum distance between any two codewords.
That is, d = min ( x,y 2 H ) ^ ( x 6 = y ) HD ( x, y ) .
The rate ofthe codeword is deﬁned as R = k/n .
Reed-Solomon code.
In this work, we will mainly use the Reed-Solomon (RS) code.
For the RS code, every message m = ( m 1 , . . .
, m k ) 2 F k is parsed as a degree k  1 polynomial
[Equation]
and the encoding is simply ( f (1) , f (2) , . . .
, f ( n )) 2 F n .
Deﬁnition 8 (List-decodable code) .
A code H ✓ F n is said to be ( p, L ) -list-decodable if for anystring x 2 F n , there are 6 L messages whose encoding c satisﬁes HD ( x, c ) 6 p · n .
We say the code
H is efﬁciently ( p, L ) -list-decodable if there is an efﬁcient algorithm that ﬁnds all such messages.
Theorem 9 (Guruswami and Sudan [1998]) .
For any constant R > 0 , the Reed-Solomon code withconstant rate R and block length n is efﬁciently (1  p R, poly ( n )) -list-decodable.
Due to space limitations, we include the preliminaries on randomness extraction in Appendix A. Thisincludes the deﬁnition of (average-case) min-entropy, samplers, etc. Additional materials on thecryptographic primitives and coding theory are also included in Appendix A.
Efﬁcient learner vs. information-theoretic learner
Consider the problem of learning an inner product function IP P deﬁned as IP P ( x ) = h x, P i , where the inner product is done in GF 2 .
Our ﬁrst observation is that to learn IP P with small error where P is uniformly random, the number of model parameters that the learner employs must be as (almost) as large as the length of P .
Intuitively, one can argue it as follows.
Let Z denote the parameters in the model that the learner outputs.
Suppose that Z is shorter than P .
Then P must contains entropy conditioned on Z .
By standard result in randomness extraction, one can argue that, for a uniform x , it holds that
[Equation]
That is, the label h x, P i looks information-theoretically uniform to the classiﬁer who holds Z and x .
Therefore, the classiﬁer can only output the correct label with (the trivial) probability ⇡ 1 / 2 .
The conclusion is that learning all linear functions needs a learner that outputs as many parametersas the function’s description is.
However, even an efﬁcient learner can do the learner just as well asan information-theoretic one (e.g., using the Gaussian elimination).
We now show how to modifythis task to make a big difference between an efﬁcient learner and an information-theoretic learner interms of the number of parameters that they output.
Computational perspective.
Now, suppose P is computationally pseudorandom [Goldwasser and
Micali, 1984] rather than being truly random.
That is, let f : { 0 , 1 }  ! { 0 , 1 } ↵ be a pseudorandom 155 generator and P is distributed as f ( U  ) , where U  denotes the uniform distribution over  bits.
Since(1) an efﬁcient learner cannot distinguish P f ( U  ) from P U ↵ and (2) any learner who tries tolearn IP P with P U ↵ needs ↵ parameters, it can be proved that an efﬁcient learner who tries tolearn IP P with P f
( U  ) must also uses ↵ parameters. However, an information-theoretic learnerneeds only  parameters to learn IP P with P f ( U  ) as it can essentially ﬁnd the seed s such that P = f ( s ) and output the seed s .
To conclude, to learn IP P for P f ( U  ) , an information-theoreticlearner only needs a few (i.e.,  ) parameters and an efﬁcient learner needs a lot of (i.e., ↵ ) parameters.
We emphasize that for a pseudorandom generator, its output length ↵ could have an arbitrarily largepolynomial dependence on its input length  .
For instance, it could be ↵ =  10 .
Robustness.
We now explain the ideas behind Theorem 32 in which we study the role of model robustness in the size of the model parameters.
We now suppose the instance is sampled according to the distribution D Q , which is deﬁned by
[Equation]
Here, U is the uniform distribution (over the right number of bits), Enc ( U ) is an error-correcting encoding of U , and the addition is coordinate-wise ﬁeld addition.
Moreover, the label for this instance is the inner product h U, P i for some vector P .
Observe that if the learner learns Q , it can always ﬁnd the correct label on a perturbed instance due to the error-correcting property of Enc ( U ) .
We argue that, in order to robustly learn this task for a uniformly random Q , the number of parameters in the model that the learner outputs must be almost as large as the length of Q .
Intuitively, the argument is as follows.
Let Z be the model that the learner outputs.
Since | Z | < | Q | , then Q contains entropy conditioned on Z .
In this case, we prove that
[Equation]
Here, ⇢ stands for the noise that the adversary adds to the instance.
In words, the classiﬁer holdingthe parameter Z cannot distinguish the perturbed instance Enc ( U ) + Q + ⇢ from a uniformly randomstring U 0 .
Since U is information-theoretically hidden to the classiﬁer, it can only output the correctlabel h U, P i with probability ⇡ 1 / 2 .
Next, to explore the difference between an efﬁcient and information-theoretic learner, we considerthe case where Q is pseudorandomly distributed, i.e. Q f ( U  ) for some f : { 0 , 1 }  ! { 0 , 1 }  .
Again, since (1) an efﬁcient learner cannot distinguish Q U  from Q f ( U  ) and (2) anylearner needs at least ⇡ Q parameters to learn the task with Q U  , an efﬁcient learner must alsoneed at least ⇡ Q parameters to learn the task.
On the other hand, an information-theoretic learnercould again ﬁnd the seed s such that Q = f ( s ) and output the seed s as the parameter.
To conclude,an information-theoretic learner requires few (i.e.,  ) parameters to robustly learn the task and anefﬁcient learner needs a lot of (i.e., ⇡  ) parameters to robustly learn the task.
(Again,  could havean arbitrary polynomial dependence on  .)
Making instances small.
The learning tasks we considered above suffer from one drawback: the size of the instance is very large, or at least is related to the number of parameters of the model.
Here we ask: is it possible to have a small instance size while an efﬁcient learner still needs to output a very large model?
We answer this question positively.
In particular, for any n = poly (  ) (e.g., n =  0 . 1 ), we construct a lerning problem where the instance size is ⇥ ( n ) and the efﬁcient learner still needs ↵ (resp.
) parameters to learn (resp.
robustly learn) the task.
Our construction uses the “sampler” by Vadhan [2003].
Informally, a sampler samp (see Theorem 20) needs a small seed u and outputs a subset of { 1 , 2 , . .
. , ↵ } of size n .
The sampler comes with the guarantee that if P is a source with high entropy, P | samp ( u ) also has high enough entropy.
To illustrate the usage of the sampler, consider learning this new inner product function IP P deﬁned as
[Equation]
For uniformly random P , one can similarly argue that a model must output at least | P | parameters to learn the task.
Let Z denote the parameters in the model that the learner outputs.
If | Z | < | P | , then P contains high entropy conditioned on Z .
By the property of the sampler, it must hold that
P | samp ( u ) contains high entropy conditioned on Z and u .
Consequently,
[Equation]
Namely, the classiﬁer who sees the parameter Z and the instance ( u, x ) can only predict the labelwith probability ⇡ 1 / 2 .
Observe that the size of the instance ( u, x ) is roughly n , 7 while P couldhave length ↵  n .
The use of the sampler in the robust learning case is similar to the non-robustcase above.
We refer the readers to Appendix B for more details.
This concludes the ﬁrst part of Theorem 1.
To sum up, we present our formal construction and formaltheorem statement below.
The formal proof is presented in the supplemental material.
Construction 10 (Parameter-heavy learning task under efﬁcient learning) .
Our construction relieson the following building blocks.
• Let f 1 : { 0 , 1 }  ! { 0 , 1 } ↵ and f 2 : { 0 , 1 }  ! { 0 , 1 }  be PRGs (Deﬁnition 4).• Let Enc be a Reed-Solomon encoding with dimension k , block length n , and rate R = k/n .The rate is chosen to be any constant < 1 / 3 and k is deﬁned by R and n .
This RS code isover the ﬁeld F 2 ` for some ` = ⇥ (log  ) .•
Let samp 1 : { 0 , 1 } r 1 !  { 1 ,..., ↵ } k · `  and samp 2 : { 0 , 1 } r 2 !  { 1 ,...,  } n · `  be samplers.
Weobtain these samplers by invoking Theorem 20 with sufﬁciently small  1 and  2 .
Forinstance, setting  1 = ⇥ (1 / log  ) and  2 to be any small constant sufﬁces.
Note that  1and  2 deﬁnes r 1 and r 2 .•
For any binary string v , we use [ v ] for an arbitrary error-correcting encoding of v (overthe ﬁeld F 2 ` ) such that [ v ] can correct > (1  R ) n/ 2 errors.
This can always be done byencoding v using RS code with a suitable (depending on the dimension of v ) rate.
Lookingforward, we shall consider an adversary that may perturb 6 (1  R ) n/ 2 symbols.
Therefore,when a string v is encoded as [ v ] and the adversary perturb it to be f [ v ] , it will always beerror-corrected and decoded back to v .We now construct the following learning task F  =
( X  , Y  , D  , H  ) .•
X  implicitly deﬁned by the distribution D  .•
Y  = { 0 , 1 } .•
D  consists of distributions D s for s 2 { 0 , 1 }  , where D s is the following.
[Equation]
u 2 ] , m, Enc ( m ) + ⇣ f 2 ( s )  samp 2 ( u 2 ) ⌘ ⌘ ,
where – u 1 and u 2 are sampled uniformly at random from { 0 , 1 } r 1 and { 0 , 1 } r 2 , respectively.
– m is sampled uniformly at random from F k 2 ` .
– f 2 ( s )  samp 2 ( u 2 ) is interpet as a vector over F 2 ` and Enc ( m ) + ⇣
f s )  samp 2 ( u 2 ) ⌘ iscoordinate-wise addition over F 2 ` .• H  consists of all functions h s : X ! Y = { 0 , 1 } for all s 2 { 0 , 1 }  , where h s is deﬁned as follow.
[Equation]
where the inner product is over F 2 and m is interpret as a string 2 F k · ` 2 in the natural way.
• Adversary.
The entire input x =
([ u 1 ] , [ u 2 ] , m, Enc ( m ) +
[Equation] ( s ) |
[Equation] [Equation]  ) is interpret as a vector over F 2 ` and we consider an adversary that may perturb 6 (1  R ) n/ 2 symbols.
That is, the distance metric is Hamming distance over F 2 ` and the adversary has a budget of (1  R ) n/ 2 .
Theorem 11.
An information-theoretic learner can (robustly) " -learn the task of Construction 10with parameter size 2  and sample complexity ⇥ ( " ) .
Moreover, an efﬁcient learner can (robustly)" -learn this task with parameter size ↵ +  and sample complexity ⇥ ( ↵ + " ) .
Theorem 12.
Any efﬁcient learner that outputs models with 6 ↵ / 2 parameters cannot " -learn F  of
Construction 10 for " < 1 / 3 .
Theorem 13.
There exists some constant c such that the following holds.
In the presence of anadversary that may perturb (1  R ) n/ 2 symbols, any efﬁcient learner for the task of Construction 10that outputs a model with c ·  / log  parameters cannot " -robustly learn F  for " < 1 / 3 .
Efﬁcient adversary vs. information-theoretic adversary
Next, we explore whether the computational efﬁciency of the adversary could affect the number of parameters required to robustly learn a task.
Previous work Garg et al.
[2020a] has considered the difference between an efﬁcient adversary and an information-theoretic one in terms of whether a successful attack can be launched.
We ﬁrst recall their construction.
The learning instance is sampled from the distribution [ vk ] , b, Sign ( sk , b ) .
Here, ( vk , sk ) is the veriﬁcation key and signing key pair of a signature scheme (Deﬁnition 6); 8 [ vk ] isan error-correcting encoding of vk , which ensures that [ vk ] can always be recovered after perturbationby the adversary; b is a uniform random bit and the label of the instance is simply b .
The signature scheme ensures that an efﬁcient adversary cannot forge a valid signature and, hence,any efﬁcient adversary that perturbs ( b, Sign ( sk , b )) will result in an invalid message/signature pair.
A classiﬁer could then detect such perturbations and output a special symbol ? indicating thatperturbation is detected.
On the other hand, an information-theoretic adversary could launch asuccessful attack by forging a valid signature (1  b, Sign ( sk , 1  b )) .Therefore
, it could perturb theinstance to be [ vk ] , 1  b, Sign ( sk , 1  b ) and, hence, ﬂipping the label of the output.
First idea.
We want to construct a learning problem such that the learner needs few parameters against an efﬁcient adversary, but a lot of parameters against an information-theoretic adversary.
Our ﬁrst idea is to add another way of recovering b in the above learning problem.
Consider the learning problem where the instance is sampled from distribution D P deﬁned as
[Equation]
Here, u is uniformly distributed.
Observe that if the learner learns P , it can always recover b correctly from [ b + h u, P i ] and [ u ] by invoking error-correction decoding.
However, if the number of parameters in the model that the learner outputs have < | P | parameters, then [ b + h u, P i ] and [ u ] information-theoretically hides b .
Again, this is because P will contain entropy conditioned on the parameters Z and, hence,
Z, u, h u, P i ⇡ Z, u, U { 0 , 1 } .
Consequently, an information-theoretic adversary could again launch the attack that replaceb, Sign ( sk , b ) with 1  b, Sign ( sk , 1  b ) and successfully ﬂipping the label.
Therefore, a learner mustemploy | P | parameters to robustly learn the task against information-theoretic adversaries.
Second idea.
In the above learning task, a learner employing | P | parameters can always recover 235 the correct label against information-theoretic adversaries.
However, against efﬁcient adversaries, alearner with fewer parameters may not always recover the correct label but will sometimes outputthe special symbol ? indicating that tampering is detected.
Could we twist it to ensure that a learnerwith fewer parameters can also always recover the correct label against efﬁcient adversaries?
Wepositively answer this question by using list-decodable code (Deﬁnition 8).
Intuitively, list-decodingensures that given an erroneous codeword, the decoding algorithm will ﬁnd the list of all messageswhose encoding is close enough to the erroneous codeword.
Our new learning task has instances drawing from the distribution D P deﬁned as
[Equation]
Here, Enc is the encoding algorithm of the list-decoding code.
The main idea is that: after theperturbation, the original message/signature pair b, Sign ( sk , b ) will always appear in the list outputby the decoding algorithm.
Now, against an efﬁcient adversary, b, Sign ( sk , b ) must be the onlyvalid message/signature pair in the list.
Otherwise, this adversary breaks the unforgeability ofthe signature scheme.
Against an information-theoretic adversary, however, it could introduce(1  b, Sign ( sk , 1  b )) into the list recovered by the list-decoding algorithm.
Consequently, thelearner cannot tell if the correct label is b or 1  b .
Consequently, for this learning task, againstinformation-theoretic adversaries, one still needs | P | parameters.
Against efﬁcient adversaries, one 250 does not need to output any parameter and can always recover the correct label.
Making instances small.
Again, we have this unsatisfying feature that the instance has the samesize as the model.
We resolve this issue using the sampler in a similar way.
We refer the readers to
Appendix C for more details.
This ﬁnishes the high-level proof of Part 2 of Theorem 1.
To sum up, we present our formalconstruction and theorem statement below.
The formal proof is presented in Appendix C.
Every instance samples a fresh ( vk , sk ) .
Construction 14 (Learning task for bounded/ unbounded attackers)
We ﬁrst introduce the buildingblocks
• ( Gen , Sign , Verify ) be a signature scheme (see Deﬁnition 6).• Let Enc be a Reed-Solomon encoding with dimension k , block length n , and rate R = k/n .We pick the rate R to be any constant < 1 / 4 and k is deﬁned by R and n .
This RS code isover the ﬁeld F 2 ` for some ` = ⇥ (log  ) .•
Let samp : { 0 , 1 } r !  { 1 ,..., ↵ } n  be samplers.
(We obtain these samplers by invokingTheorem 20 with sufﬁciently small  1 and  2 .
For instance, setting  1 = ⇥ (1 / log  ) and 2 to be any small constant
sufﬁces.)• For any binary string v , we use [ v ] for an arbitrary error-correcting encoding of v (overthe ﬁeld F 2 ` ) such that [ v ] can correct > (1  p R ) n errors.
This can always be done byencoding v using RS code with a suitable (depending on the dimension of v ) rate.
Lookingforward, we shall consider an adversary that may perturb 6 (1 p R ) n symbols.
Therefore,when a string v is encoded as [ v ] and the adversary perturb it to be f [ v ] , it will always beerror-corrected and decoded back to v .
We now construct the following learning task F  =
( X  , Y  , D  , H  ) .
• X  is { 0 , 1 } N for some N that is implicitly deﬁned by D  .•
Y  is { 0 , 1 } .•
The distribution D  consists of all distribution D s for s 2 { 0 , 1 } ↵ , where
[Equation]
Here,– u is uniformly at random from { 0 , 1 } r and v is uniformly at random from { 0 , 1 } n .
– ( vk , sk ) Gen (1  ) are sampled from the signature scheme.
– b is sampled uniformly at random from { 0 , 1 } .
( b, Sign ( sk , b )) is intepret as F k 2 ` in thenatural way.•
h  consists of one single function h .
On input x =
([ u ] , [ v ] , Enc ( b, Sign ( sk , b ) , [ b +h v, s | samp ( u ) i ]) , h ( x ) simply decode Enc ( b, Sign ( sk , b )) and output b .
• Adversary.
The entire input ([ u ] , [ v ] , [ vk ] , Enc ( b, Sign ( sk , b )) , [ b + h v, s | samp ( u ) i ]) is interpret as a vector over F 2 ` and we consider an adversary that may perturb 6 (1  p R ) nsymbols.
That is, the distance metric is Hamming distance over F 2 ` and the adversary has abudget of (1  p R ) n .
Theorem 15.
For the learning task of Construction 14, there is an efﬁcient learner (with 0 samplecomplexity) that outputs a model with no parameter and negl (  ) -robustly learns F  against efﬁcientadversaries of budget (1  p R ) n .
Theorem 16.
Consider the learning task of Construction 14. Then, for computationally unboundedadversaries, any information-theoretic learner with ↵ / 2 parameters cannot " -robustly learn F  for" < 1 / 3 . Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes]
[Table]
(c) Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A](b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A](d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[Yes](c)
Did you include any new assets either in the supplemental material or as a URL?
[No](d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]