Paper: On robust overfitting: adversarial training induced distribution matters (ICLR-2024)

Reviewer comment (Theory):
This paper seems trying to persuade readers that the proposed "local dispersion" plays an important role (even could be the cause) in robust overfitting, which however is misleading. Actually, this paper only EMPIRICALLY showed that there seems to be a correlation between local dispersion and robust overfitting, but unfortunately neither provided any theoretical explanation about the correlation nor explained how one can understand robust overfitting through this correlation. From this perspective, I would not treat "local dispersion" as a "contribution" of this paper.

Author response:
The observation that local dispersiveness correlates with robust overfitting is for the first time reported. In this respect, we believe it is a contribution. The bound provided in our Theorem 1 provides an theoretical explaination for understanding robust overfitting through local dispersion. Specifically, the bound suggests that local dispersiveness of the perturbation operators makes the induced distribution harder to learn, a phenomenon correlated well with robust overfitting.

Reviewer comment (Theory):
The authors put a lot of effort into theoretically and empirically explaining the "generalization difficulty of the induced distribution", which however is weird. According to the paper itself, this "generalization difficulty" is only a result of robust overfitting, but could not be used to explain robust overfitting itself. In other words, the "generalization difficulty" as well as the "induced distribution" do not matter to robust overfitting (although the authors said they matter according to the title of this paper).

Author response:
We did observe correlation between generalization difficulty of the induced distribution and robust overfitting. However, based on this correlation, claiming "robust overfitting causes geneneralzation difficulty" or claiming the oppositve direction of causality are equally ungrounded.

Reviewer comment (Novelty):
Not a single paper from distributionally robust optimization is cited. For years there have been many variants of Theorem 1, bounding the deviation of the empirical risk from the true risk under the assumption that the training set is sampled i.i.d. from a data-generating distribution. There are quite strong results out there. Not only do these works derive upper bounds on the risk deviations, but they also develop and solve distributionally robust counterparts of adversarial training.

Author response:
We agree that adversarial training is related the DRO literature, and we have included some of the DRO works in our section of related works in the revision. We must point out however that, to the best of our knowledge, all analyses of generalization in the DRO literature, including the ones the reviews pointed to, ignore the impact of training dynamics on the generalization behaviour of the learned models. As discussed earlier, such an approach often fails to be relevant to deep learning systems. The two papers (Kuhn et al. (2019) and Esfahani and Kuhn (2018)) pointed to by the reviewer are in fact perfect demonstration of the limitation of this approach.

Specifically, Theorem 18 in "Wasserstein distributionally robust optimization: Theory and applications in machine learning" by Kuhn et al. (2019) and Theorem 3.4 in "Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations" by Mohajerin Esfahani and Kuhn, 2018) are much worse than the bound we provide in Theorem 1 of this paper. To see this, consider CIFAR 10 dataset, in which the input dimension is 3072. Acccording the bounds in Kuhn et al. (2019) and Esfahani and Kuhn (2018), high-probability generalization error decays at the rate of $m^{-1/3072}$, whereas in our bound, the error decays as $m^{-1/2}$, where $m$ is the size of the training sample (which was denoted by $N$ in their papers). In fact, it is precisely because these bounds are algorithm-agnostic, they insist that regardless of the loss function being the standard loss $\ell$ or the adversarial loss $\ell^{\epsilon}$ (as you defined), the generalization error decays at the same rate, hence unable to explain the drastic difference in generalization between AT and standard training.   

Nonetheless, thank you for pointing to the connection to the DRO literature. In light of this connection,  the approach developed in this paper may in fact translate to additional insights into the study of the DRO problem.

Reviewer comment (Theory):
The focus of this paper (and most cited related work) is on the structure of overfitting as a result of iterations of a specific training algorithm, such as PGD-AT, where parameter updates and adversarial attacks iteratively influence the induced data distribution; this requires a clearer discussion.

Author response:
Thank you very much for your careful review.  We believe that the reviewer has misunderstood the main point of our work. 

It is correct, as you stated, that "robust overfitting is a result of overfitting the adversarially induced data" (we will discuss this statement in detail later). It is also correct that one may consider adversarial training as training with a modified loss function and apply standard learning-theoretic analysis to analyze the generalization behaviour of adversarial training. However, such a view is fundamentally limited, since it fails to explain why adversarily trained model is much more prone to overfitting, for example on CIFAR10, than models trained on the clean dataset (using the standard loss). This paper studies the generalization behaviour in adversarial training by zooming into the dynamics of PGD-AT iterations and attempts to understand the cause leading to the final induced distribution difficult to generalize. Such a philosophy, i.e., tracking the training dynamics and understanding its impact on the generalization behaviour of the learned model, has demonstrated great successes in deep learning research, for example, in Arora et al, "Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks",  in Soudry et al, "The Implicit Bias of Gradient Descent on Separable Data",  in Lyu et al, "Gradient Descent Maximizes the Margin of Homogeneous Neural Networks" and in various works on Neural Tangent Kernel (Jacot, 2018). Such algorithm-dependent analysis has proven to be much more relevant to deep learning models than the pessimistic bounds given by classical learning theory, which puts primary emphasis on the loss function and the hypothesis class. 

The reviewer considers that the key message of our paper is "robust overfitting is a result of overfitting the adversarially induced data" and that we merely "have rediscovered" this. We like to note: 1) this is not our key message, 2) the presented results are much more than rediscover this.  First, when speaking of "robust overfitting is a result of overfitting the adversarially induced data", the adversarially induced data (distribution) refers to the distribution induced by adversarial perturbation against the final learned model, or, using the notation of this paper, it refers to the distribution $\tilde{{\cal D}}_T$ at final iteration $T$. Empirical results have shown that the learned model from PGD-AT does not generalize well with respect to $\tilde{{\cal D}}_T$, hence the conclusion "robust overfitting is a result of overfitting the adversarially induced data". This phenomenon requires no re-discovery. The main question motivating this research is why $\tilde{{\cal D}}_T$ becomes difficult to generalize. We thus set out to investigate how the perturbation-induced distributions $\tilde{{\cal D}}_t$ evolve along the trajectory of PGD-AT and the factors that may cause $\tilde{{\cal D}}_t$ to be increasingly difficult to generalize.

Reviewer comment (Novelty):
Given that adversarial training is essentially ERM with a modified loss incorporating adversarial perturbations, standard overfitting theory applies; thus, the claimed "induced distribution overfitting" is essentially classical overfitting, making the discovery less interesting.

Author response:
We understand the reviewer's perspective that regards robust overfitting as the classical notion of overfitting under a different choice of loss function. However, classifical learning-theoretic bounds involve complexity measures (e.g. Rademarcher Complexity) that are difficult to characterize for deep neural networks and one must recourse to upper bounds of the complexity measures, making the bounds too loose to be relevant. This difficulty aggravates in AT when the loss function involves maximization over a norm ball, i.e., $\ell^{\epsilon}$.  Our approach unfolds the training dynamics, and enables us to obtain quantities (i.e. local dispersion) that closely track the model's generalization behaviour as it is updated along the training trajectory.

Reviewer comment (Theory):
Robust overfitting, like classic overfitting, arises from the optimizer's curse—parameters are optimized over a sample set \( S \) drawn from the true distribution \(\mathcal{D}\), causing a nonzero gap; thus, techniques from distributionally robust optimization (DRO) are directly relevant.

Author response:
We agree that adversarial training is related the DRO literature, and we have included some of the DRO works in our section of related works in the revision. We must point out however that, to the best of our knowledge, all analyses of generalization in the DRO literature, including the ones the reviews pointed to, ignore the impact of training dynamics on the generalization behaviour of the learned models. As discussed earlier, such an approach often fails to be relevant to deep learning systems. The two papers (Kuhn et al. (2019) and Esfahani and Kuhn (2018)) pointed to by the reviewer are in fact perfect demonstration of the limitation of this approach.

Specifically, Theorem 18 in "Wasserstein distributionally robust optimization: Theory and applications in machine learning" by Kuhn et al. (2019) and Theorem 3.4 in "Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations" by Mohajerin Esfahani and Kuhn, 2018) are much worse than the bound we provide in Theorem 1 of this paper. To see this, consider CIFAR 10 dataset, in which the input dimension is 3072. Acccording the bounds in Kuhn et al. (2019) and Esfahani and Kuhn (2018), high-probability generalization error decays at the rate of $m^{-1/3072}$, whereas in our bound, the error decays as $m^{-1/2}$, where $m$ is the size of the training sample (which was denoted by $N$ in their papers). In fact, it is precisely because these bounds are algorithm-agnostic, they insist that regardless of the loss function being the standard loss $\ell$ or the adversarial loss $\ell^{\epsilon}$ (as you defined), the generalization error decays at the same rate, hence unable to explain the drastic difference in generalization between AT and standard training.   

Nonetheless, thank you for pointing to the connection to the DRO literature. In light of this connection,  the approach developed in this paper may in fact translate to additional insights into the study of the DRO problem.

Reviewer comment (Theory):
The paper discusses concepts (data shifts, overfitting) that are central to DRO, but fails to connect to the established results and methodology in that field, such as the Wasserstein-Kantorovich duality and tighter bounds available in the literature.

Author response:
We agree that adversarial training is related the DRO literature, and we have included some of the DRO works in our section of related works in the revision. We must point out however that, to the best of our knowledge, all analyses of generalization in the DRO literature, including the ones the reviews pointed to, ignore the impact of training dynamics on the generalization behaviour of the learned models. As discussed earlier, such an approach often fails to be relevant to deep learning systems. The two papers (Kuhn et al. (2019) and Esfahani and Kuhn (2018)) pointed to by the reviewer are in fact perfect demonstration of the limitation of this approach.

Specifically, Theorem 18 in "Wasserstein distributionally robust optimization: Theory and applications in machine learning" by Kuhn et al. (2019) and Theorem 3.4 in "Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations" by Mohajerin Esfahani and Kuhn, 2018) are much worse than the bound we provide in Theorem 1 of this paper. To see this, consider CIFAR 10 dataset, in which the input dimension is 3072. Acccording the bounds in Kuhn et al. (2019) and Esfahani and Kuhn (2018), high-probability generalization error decays at the rate of $m^{-1/3072}$, whereas in our bound, the error decays as $m^{-1/2}$, where $m$ is the size of the training sample (which was denoted by $N$ in their papers). In fact, it is precisely because these bounds are algorithm-agnostic, they insist that regardless of the loss function being the standard loss $\ell$ or the adversarial loss $\ell^{\epsilon}$ (as you defined), the generalization error decays at the same rate, hence unable to explain the drastic difference in generalization between AT and standard training.   

Nonetheless, thank you for pointing to the connection to the DRO literature. In light of this connection,  the approach developed in this paper may in fact translate to additional insights into the study of the DRO problem.

Reviewer comment (Novelty):
The omission of DRO literature is concerning because these works offer stronger bounds than Theorem 1 and contain algorithms to mitigate overfitting, which directly relates to the paper’s stated goals and could limit its contribution.

Author response:
We agree that adversarial training is related the DRO literature, and we have included some of the DRO works in our section of related works in the revision. We must point out however that, to the best of our knowledge, all analyses of generalization in the DRO literature, including the ones the reviews pointed to, ignore the impact of training dynamics on the generalization behaviour of the learned models. As discussed earlier, such an approach often fails to be relevant to deep learning systems. The two papers (Kuhn et al. (2019) and Esfahani and Kuhn (2018)) pointed to by the reviewer are in fact perfect demonstration of the limitation of this approach.

Specifically, Theorem 18 in "Wasserstein distributionally robust optimization: Theory and applications in machine learning" by Kuhn et al. (2019) and Theorem 3.4 in "Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations" by Mohajerin Esfahani and Kuhn, 2018) are much worse than the bound we provide in Theorem 1 of this paper. To see this, consider CIFAR 10 dataset, in which the input dimension is 3072. Acccording the bounds in Kuhn et al. (2019) and Esfahani and Kuhn (2018), high-probability generalization error decays at the rate of $m^{-1/3072}$, whereas in our bound, the error decays as $m^{-1/2}$, where $m$ is the size of the training sample (which was denoted by $N$ in their papers). In fact, it is precisely because these bounds are algorithm-agnostic, they insist that regardless of the loss function being the standard loss $\ell$ or the adversarial loss $\ell^{\epsilon}$ (as you defined), the generalization error decays at the same rate, hence unable to explain the drastic difference in generalization between AT and standard training.   

Nonetheless, thank you for pointing to the connection to the DRO literature. In light of this connection,  the approach developed in this paper may in fact translate to additional insights into the study of the DRO problem.

Reviewer comment (Writing):
In the abstract, several terms are unclear until later definitions are given (e.g., "evolution of the data distribution," "checkpoints," "certain local parameters"), and there is circular reasoning in statements about robust overfitting.

Author response:
- In the abstract, the terms used will not be clear until one reads the paper. For example "evolution of the data distribution" will be defined later on and here it does not make meaning. Similarly "checkpoints", or "certain local parameters" are unclear yet. Also in the abstract, there is circular reasoning: initially, it is said this paper will explain why robust overfitting happens, but then "We observe that the obtained models become increasingly harder to generalize when robust overfitting occurs" -> Aren't we investigating the reason behind robust overfitting? 

We've considered your suggestion and completely rewritten the abstract.

Reviewer comment (Theory):
"robust population risk \(R_{\mathcal{D}}^{\text{rob}}\)" is defined incorrectly; it is actually adversarial risk, and the paper uses "robust" and "adversarial" interchangeably, which should be corrected.

Author response:
- "robust population risk $R_{\mathcal{D}}^{\text{rob}}$" is defined wrongly. This is adversarial risk. Robust refers to the model that is trained adversarially robust. The definition the authors provide is for any $\theta$, regardless of how they are trained. Overall, the paper uses "robust" and "adversarial" interchangeably, and this should be corrected. 

We've modified the "robust population risk" as "adversarial population risk", and denote it as $R^{\rm adv}_{\mathcal{D}}(\theta)$.

Reviewer comment (Theory):
The statement "one may expect the generalization performance of a model trained on \(\mathcal{D}_t\) is similar to that on \(\mathcal{D}\)" contradicts existing work on distributional shifts and DRO.

Author response:
- "As such, one may expect the generalization performance of a model trained on $\mathcal{D}_t$ is similar to that on $\mathcal{D}$" would contradict every paper that is concerned with distributional shifts and DRO.

The sentence was not structured correctly, and is now removed.

Reviewer comment (Writing):
Some terms are used before they are defined or abbreviated, such as "PGD" being mentioned before definition in Section 3.

Author response:
- Some terms are used before they are defined or abbreviated. Examples: "PGD" name is being used in the beginning before it is defined in S3.

We've fixed this in our paper.

Reviewer comment (Writing):
There is a sense of mystery in the discussion; for example, describing overfitting as "unexpected" without justification.

Author response:
- There is some mystery behind discussions. In the beginning, for example, overfitting is said to be "unexpected" -> why? 

In that sentence, we do not mean that overfitting is unexpected. What is meant is that much more severe overfitting in adversarial training compared to standard training is unexpected.

Reviewer comment (Writing):
Some statements are overly informal, e.g., "A *great deal* of research effort has been spent" without references, and colloquial questions such as "does robust overfitting have anything to do with this."

Author response:
- Some statements are informal. Examples: "A *great deal* of research effort has been spent" is not followed by any references. On the next page, "does robust overfitting have anything to do with this" is also informal. 

We have revised this paragraph

Reviewer comment (Writing):
On page 1, terms like "smoothing logits" are not clear; when citing other works, the paper should not assume familiarity with their specific terminology.

Author response:
- Page 1, "smoothing logits" -> not clear. In general, when a paper is cited, please make sure to not assume the reader knows the specific terminology.

We have improved the writting here.

Reviewer comment (Writing):
The paragraph at the end of page 2 refers to aspects specific to the training method, lacking broader generality.

Author response:
- The paragraph at the end of page 2: Again, this is specific to the training method.

We have revised this paragraph

The experimental observation and the related conclusion in our work are all towards PGD-AT, this specific training method. We've added a few line of statements in our paper to clarify this point.

Reviewer comment (Writing):
The claim that "the most popular adversarial training" is PGD-AT is limited to certain learner classes, such as neural networks.

Author response:
- "The most popular adversarial training" is said to be PGD-AT, but this is specific to a class of learners, e.g., neural networks.

We have revised the wording to restrict the context to neural networks.

Reviewer comment (Novelty):
The description of PGD-AT from the end of page 3 onward should include citation to a generic reference.

Author response:
- Could the authors cite some generic paper for the PGD-AT as described from the end of page 3 onward? 

We have fixed this following your suggestion.

Reviewer comment (Theory):
PGD-AT provides only an approximate solution, yet conclusions are drawn for adversarial learning in general; the distinction should be clarified.

Author response:
- The described PGD-AT provides an approximate solution, not necessarily a strong solution. However, the conclusions drawn in this paper are for adversarial learning in general. This distinction could be clarified. 

We've clarified in the paper that our research focus on PGD-AT with deep neural networks instead of adversarial learning in general.

Reviewer comment (Theory):
Section 6 is heuristic and informal; without stronger formal foundations in preceding sections, it reduces the paper’s overall formality.

Author response:
- Section 6 is overall a heuristic approach, and informal. If the preceding sections were formal enough, this would be a good teaser, but at the current form, I think it makes the paper less formal.

We have revised the section to improve its formality.

Reviewer comment (Writing):
In the introduction, the term "error probability in the predicted label for adversarially perturbed instances" raises the question of whether "probability" is the correct word.

Author response:
- Introduction: "error probability in the predicted label for adversarially perturbed instances" -> Is "probability" the correct word?

We think "error probability" is an acceptable phrase in this context.

Reviewer comment (Theory):
For clarity, the paper should mathematically specify what each "error" looks like, discuss parts of proofs that would fail for \(\ell_{p \neq \infty}\) norm attacks, and explain the interest in other \(\ell_p\) attacks.

Author response:
- For clarity, can the authors specify what each "error" looks like, maybe via mathematical notation? In the above sentence, for example, "the error in the predicted label for adversarially perturbed instances" is mentioned, but the structure of adversarial attacks comes later. That said, I was also wondering, can the authors add a sentence discussing what part of their proofs would not work in the case of ℓp≠∞ norm attacks? Finally, the ℓp attacks, even when they are not restricted to ∞-norms, typically deserve a quick discussion on why such attacks are interesting. 

Should the reviewer insist that we insert a mathematical equation defining the error and restructure description in the introduction, we will do that in the final revision. But we think, for the introduction, it might be better to stay at a high level and defer the precise explanation to the body of the paper.

Reviewer comment (Writing):
On page 2, the statement that Dong et al. and Yu et al. conflict uses incorrect grammar ("conflict 'to'" instead of "conflict 'with'") and requires clarification on how their explanations differ; also, "each shown to" is probably a typo.

Author response:
- Page 2 says Dong et al and Yu et al. conflict 'to' each other (conflict "with" might perhaps be the better word): could the authors please explain a little bit about how these explanations vary? The sentence that follows says "each shown to" which probably is a typo.

This is now fixed.

Reviewer comment (Writing):
Before introducing PGD training, the phrase "along adversarial training" may mislead; this should be clarified at the start, as should the meaning of "induces a new data distribution" and terminology like "training step \(t\)."

Author response:
- Before the PGD training is mentioned, the paper uses the phrase "along adversarial training". I think the "along" implies that the algorithm is iterative and again it is specific to the structure of the PGD-AT algorithm. Could the authors please mention this at the beginning of the paper? Similarly, after that "induces a new data distribution" is not immediately clear; I only understood what is meant here after I read the paper fully once. Please also clarify "at training step t" terminology. 

We have revised this part to improve clarity.

Reviewer comment (Presentation):
Before Figure 1, including a step-by-step framework of IDE testing would improve readability, as the intended procedure only becomes clear in Section 4.

Author response:
- Right before Figure 1, could the authors please include the framework of IDE testing step by step? It does not need to be long but it would improve the readability. I understood the whole story much better in the first paragraph of Section 4.

We have revised this part.

Reviewer comment (Presentation):
WRN-34 and PRN-18 should be cited when first mentioned.

Author response:
- Can the authors please cite WRN-34 and PRN-18 when they first appear? 

This has been fixed.

Reviewer comment (Theory):
At the end of Section 2, the paper should elaborate on the "mismatch between theoretical settings and real-world practices."

Author response:
- End of Section 2: can you please elaborate on the "mismatch between theoretical settings and real-world practices"

This remark is now deleted.

Reviewer comment (Writing):
On page 4, the meaning of "**distribution** of \((v,y) = ...\)" should be clarified; is it randomness from \(\rho\) or the empirical collection of perturbed training instances?

Author response:
- Could the authors please clarify what on page 4 "**distribution** of (v,y)=..." refers to? Is it the distribution whose randomness comes from ρ? Or do the authors mean the collection of training instances perturbed with respect to the underlying mechanism? 

We have added the phrase "random variable pair" before $(v, y)$ so that it is now clear that the distribution refers to their joint distribution.

Reviewer comment (Writing):
The term "checkpoints" might be confusing; "IDE is recorded every \(k\) iterations" could be clearer.

Author response:
- I guess the name "checkpoints" is coined because it is expensive to do this IDE check at every iteration. Would it perhaps be less confusing if we do not mention checkpoints, rather say "IDE is recorded every k iterations"? 

We have now better explained what checkpoints are referred to.

Reviewer comment (Experiments):
Experiments are repeated only five times; this small number may conflict with stochastic optimization literature regarding the importance of \(\rho\) selection; also, the choice of 10 pairs of \(\rho, \rho'\) needs discussion.

Author response:
- The experiments are repeated only 5 times, isn't this also contradicting the stochastic optimization literature/curse of dimensionality etc., as the choice of ρ is crucial? Maybe the SAA literature has an answer to this. (Similarly the 10 pairs of ρ,ρ′ need discussion).

Agreably, more repeatitions will improve our estimation. We will make an effort in that direction in the final revision.

Reviewer comment (Presentation):
Figure 3(a) should be replotted over the error rate range [0, 0.1] for better readability.

Author response:
- Can Figure 3 (a) be plotted over the error rate range [0, 0.1]? It is hard to read on this scale. 

This choice of scale is for an easy visual comparison with Figure 3 (b).  The exact error values in 3(a) do not matter as much, since key message is to show its difference wtih 3(b).

Reviewer comment (Theory):
In Section 5, using \(f \in \mathcal{F}\) instead of explicit adversarial attack notation should be justified; it should be clarified if the proof holds for \(\varepsilon = 0\) and what properties of \(f\) relate to adversarial training, as the boundedness of perturbation-smoothed loss may be too loose.

Author response:
- Is there a reason why Section 5 uses f∈F notation rather than keeping the adversarial attacks explicit? Would the proof make sense if ε=0 as well? Also, the f∈F refers to **any** function, can the authors make it clear how the adversarial setting is used in the proof? I guess the only property of f about the adversarial training is the "boundedness of perturbation-smoothed loss) -- isn't this very loose?

 We wish to note that the proof is not directly on adversarial training, rather it is proving a generalization bound for IDE experiments.

Reviewer comment (Theory):
The term "measurable" should be defined with respect to the relevant \(\sigma\)-algebra, since the norm space is explicitly defined.

Author response:
- "measurable" -> With respect to which algebra? (I am asking because the norm space is explicitly defined).

Here measurable is defined with respect to the standard Borel algebra.

Reviewer comment (Evaluation):
The results in section 6 on the dispersion of perturbations along the adversarial training process, although interesting, do not fit well into the main message of the paper. In fact, the results of experiments on the evolution of $d_\theta(x, y)$ seem to undercut the main message.

Author response:
Thank you very much for your careful review. Regarding the weakness that you pointed here, we will make an effort to revise section 6 to improve its presentation in the final revision. We were not able to do it during the rebuttal period, due to too much time spent on addressing a concern of another reviewer 3NwP.

Reviewer comment (Theory):
Why is $\rho$ in (5) taken to be from the uniform distribution on the hypercube as opposed to any other distribution centered at $x$ and supported on $B(x, \epsilon)$?

Author response:
1. We have reformulated the setting and now define the data distribution ${\cal D}$ as resulting from uniformly perturbing a more robust distribution ${\cal D}^{*}$. For this reason, $\rho$ drawn uniformly at random from the $\epsilon$-ball naturall arise. Please refer to Section 5 for our reformulation.

Reviewer comment (Evaluation):
Using only 10 pairs of $(\rho, \rho')$ in the Monte-Carlo estimate of $\gamma_{B(x, \epsilon)}(Q_{x, y, \theta})$ seems awefully inadequate especially because the hypercube $B(x, \epsilon)$ is of large dimension. Typically, the number of samples you need to estimate an expectation over a hypercube increases exponentially with dimension. I have the same comment about the estimate of $d_\theta(x,y)$. What is the variance you observe for these estimates?

Author response:
3. We have improved the experiments by sampling 250 pairs in the Monte-Carlo estimate. These experiments requires significant computing resources. In this new experiment, the variance of the estimate ranges from 0.0025 to 0.0166 at various checkpoints. (noting the estimate of ELD is about 0.43 to 0.6). Although this experiments can be improved by increasing the number of samples, we believe that the message from this experiments is already convicing. In the final revision, we will make an effort to further increase the number of samples in this estimation.

Reviewer comment (Experiments):
Is there a reason for restricting the adversarial perturbations to $L_\infty$ norm only, as opposed to keeping it general to any norm?

Author response:
4. The main focus of this paper is to develop improved understanding of robust overfitting. Since in PGD-AT defined using other norms, robust overfitting does not arise as severely. This has been the reason that this paper is restricted only to infinity-norm based PGD-AT. Nonetheless, it may also be interesting to look into other kinds of perturbations to develop more general understanding of robust overfitting.