From: NeurIPS.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept

From: NeurIPS.cc/2022/Conference/Paper11065/Area_Chair_Nq4A
metareview: This paper studies the computational benefits of overparametrization in standard and robust learning. The paper suggests that there are learning tasks where overparametrization is needed from a computationally efficient, standard and robust, learning.
In Theorem 1 Part 2, the paper shows that there are robust learning tasks such that there are efficient underparameterized learners against computationally bounded adversaries, but overparamertization is needed when the adversary is computationally unbounded (even for information-theoretically efficient robust learners).
The result indicates that there is a statistical-computational tradeoff between efficient robust learning and inefficient robust learning.

The referees are leaning towards acceptance and that is also my recommendation,
recommendation: Accept
confidence: Less certain
award: No

From: NeurIPS.cc/2022/Conference/Paper11065/Authors
title: Thank you.
comment: Dear reviewer 7oiG, 

Thank you for raising this interesting point. We added a discussion on this in appendix E. There, we mention that our work does not consider real valued parameters with infinite precision because computationally bounded machines are not able to store or read them. We will move this discussion to the main body in the camera-ready version.

From: NeurIPS.cc/2022/Conference/Paper11065/Authors
title: new section added
comment: Dear reviewer WAyy,

Thank you for your suggestion. We added a new discussion in appendix E reflecting on implications and limitations of our work. We included this in appendix due to space limitations. We will include this in the main body of our paper in the potential camera-ready version, which allows for an additional page.

From: NeurIPS.cc/2022/Conference/Paper11065/Reviewer_WAyy
title: Thank you.
comment: Thank you for addressing my question. I strongly suggest that the authors attempt to discuss practical implications in future revisions of their work.

From: NeurIPS.cc/2022/Conference/Paper11065/Authors
title: To all reviewers
comment: We appreciated the thoughtful review, and are available to answer any questions.

From: NeurIPS.cc/2022/Conference/Paper11065/Reviewer_7oiG
title: Respone
comment: Thank you for your response. 

I did not realize that the efficient learning of real parameters is "undefined". It sort of answers the question; the point was that 1 real parameter is indistinguishable from any countable set of real parameters (bijection between $\mathbb{R}$ nd $\mathbb{R}^\omega$).

From: NeurIPS.cc/2022/Conference/Paper11065/Authors
title: Response to Reviewer Sv39
comment: Thanks for pointing out the issues regarding missing definitions and the ordering of theorems. We shall be happy to fix them and make appropriate modifications to ensure that the theorems we cite appear in the main paper.
 
Our rationale for including the formal constructions and theorem statement in the main body is to give a formal treatment of the intuitive claims we make during the high-level overview.

The techniques we use in non-robust learning (pg 5, 154-164) and robust learning (pg 6, 169-177) are indeed similar. Note that we have the freedom to pick any alpha and beta (as an arbitrary polynomial in lambda) we want. Our construction chooses to pick beta >> alpha. This construction demonstrates our point that there is a learning problem such that a robust efficient learner needs more parameters (i.e., alpha + beta) than a non-robust efficient learner (i.e., alpha). One could also imagine the case where beta < alpha. However, it would give a learning problem where the efficient robust learner and efficient non-robust learner require a similar number of parameters, which is not very interesting to us.

From: NeurIPS.cc/2022/Conference/Paper11065/Authors
title: Response to Reviewer WAyy
comment: For theorem 11, the reviewer is correct that the efficient learner does need (multiplicatively) polynomially more samples than an information-theoretic learner. The construction we have in this paper does not give such an example where the efficient learner only requires more parameters but the same amount of samples. We believe this is an excellent question and exciting future work.

From: NeurIPS.cc/2022/Conference/Paper11065/Authors
title: Response to Reviewer 7oiG
comment: In this work, we consider the simplest setting of binary output and construct such a learning problem where there is an arbitrary polynomial gap in the number of parameters that an efficient learner and an information-theoretic learner require. One can extend it to more sophisticated settings of any finite output range, particularly real numbers with bounded precision. However, if one considers real numbers with infinite precision, it is unclear what it means for an algorithm to be computationally efficient (as it needs to take an infinitely long input). Hence, we do not consider it in this work.

We are not sure if we completely understand what the reviewer means by “ the number of parameters needs to be reconsidered here due to space filling curves.” We are happy to clarify more if the reviewer could elaborate or has more questions.

From: NeurIPS.cc/2022/Conference/Paper11065/Reviewer_7oiG
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
summary: The paper considers the tasks of adversarial robustness and explores the difference between omniscient adversary, and the computationally bounded adversary. It constructs several learning tasks demonstrating the difference  between informatic theoretic and efficient learning.
strengths_and_weaknesses: I did not find anything wrong with the paper, but I haven't understood its arguments either due to my limited knowledge of this area.
questions: Do the results work also for infinite (and in particular, uncountable) domains? Such as learning something from real-numbered observations? Specially the number of parameters needs to be reconsidered here due to space filling curves.

I'm sorry for such an uninformative review; it did not match my knowledge and I don't have the time to understand the relevant context.
limitations: -
ethics_flag: No
soundness: 2 fair
presentation: 2 fair
contribution: 2 fair

From: NeurIPS.cc/2022/Conference/Paper11065/Reviewer_WAyy
rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: This paper studies the computational benefits of overparametrization in standard and robust learning. The paper suggests that there are learning tasks where overparametrization is needed from a computationally efficient, standard and robust, learning. 

More specifically, as I understand it,  the paper shows in Theorem 1 (Part 1) that there are learning tasks where “efficient” standard learning and “efficient” robust learning require polynomial (in the input/instance size) overparameterization. 

In Theorem 1 Part 2, the paper shows that there are robust learning tasks such that there are efficient underparameterized learners against computationally bounded adversaries, but overparamertization is needed when the adversary is computationally unbounded (even for information-theoretically efficient robust learners). 

The result indicates that there is a statistical-computational tradeoff between efficient robust learning and inefficient robust learning.

strengths_and_weaknesses: I find the main result of the paper interesting. The gaps established in this work are polynomial (as opposed to exponential), but I think that’s still interesting from a practical perspective. The paper is rigorous and formal in its treatment of the definition of “efficient” learning, which is nice. The paper is clear and well-written.


questions: In Theorem 1 Part 1, by checking Theorem 11, it seems that the sample complexity is also polynomially larger. I wonder if it is possible to obtain a result where the efficient learner only needs overparamterization but no increase in sample complexity compared to the information-theoretic learner.

limitations:  It might be good to reflect on the results of the paper, and discuss their practical implications, and any future directions that might be good to address from this work.

ethics_flag: No
soundness: 4 excellent
presentation: 3 good
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper11065/Reviewer_Sv39
rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
confidence: 1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
summary: This work studies whether there exist any learning tasks that are computationally bounded robust learning need to have more model parameters than non-robust learning; and whether such tasks can be done with fewer model parameters for polynomial-time adversaries compared with information-theoretic adversaries. The author gives positive answers to both of the questions by leveraging cryptography and coding theory.
strengths_and_weaknesses: The questions asked by the work are interesting and novel to the best of my knowledge. Such studies would be helpful for the robust adversarial learning community. 
The writing needs to have more structure improvement. However, since I’m not familiar with cryptography and coding theory, I’m not able to judge whether the technical idea is novel or sound.
questions: 1. Missing formal definition of $L(\cdot,\lambda)$.
2. In the main paper top of page 6 you discuss theorem 32, which is a restatement of theorem 13, and would be good to state theorem 13 before discussing it. In the statement of construction 10, you mention theorem 20 which also does not appear in the main paper. 
3. Personally I did not find it’s helpful to give construction 10 and 14 in detail in the main paper.  It’d be helpful to combine theorem 11, 12, 13, 15, 16 with their discussion or comments instead of isolating them separately.
4. It seems to me that the argument of computational perspective on page 5 line 154-164 has a similar idea as the robustness on page 6 line 169-177, and I did not find any arguments regarding why $\alpha<\beta$, but this should be an important criterion in theorem 1. Can you claim what would happen if I replace $\beta$ with $\alpha$ in argument line 169-177?

limitations: Yes
ethics_flag: No
soundness: 3 good
presentation: 2 fair
contribution: 3 good