Paper: A Sublinear Adversarial Training Algorithm (ICLR-2024)

Reviewer comment (Novelty):
The relevance of the paper remains unclear. Is it theoretically relevant or also practically?

Author response:
Our paper is theoretically relevant and analyzes the convergence guarantee of adversarial training procedures on a two-layer neural network with shifted ReLU activation, demonstrating that only a sublinear number of neurons (o(m)) are activated for each input data per iteration​ and developing an algorithm for adversarial training with time cost o(mnd) per iteration by applying half-space reporting data structure.

Reviewer comment (Presentation):
The paper is hard to read, and the reasoning jumps back and forth. The reviewer recommends a thorough rewrite. It would be good if the authors first introduce concepts and then use them. An example is the "query, insert, remove" from the data structure in Algorithm 1 (Page 6) that gets introduced in Section 5 (Page 8). Similarly, Theorem 1.1 (Page 2) has a reference to Equation 2 (Page 4). The paper seems rushed and reads that way.

Author response:
We thank you for pointing out the writing improvement suggestions. We will update the overall structure in the next version.

Reviewer comment (Novelty):
Are the contributions of the paper mainly of theoretical or also practical interest?

Author response:
The contribution of this paper is a theoretical. The introduced half-space reporting data structure used in this paper will take additional $O(n \log n)$ space [12] . To have an intuitive understanding of the level of such additional memory usage, we would point out that the weights of neural networks takes $O(n d)$ space. So when the neural network is over-parameterized, i.e., $d$ is very large, the additional memory usage by our algorithm is not the dominant term in space complexity.

Reviewer comment (Reproducibility):
What are the memory requirements of the algorithm?

Author response:
The contribution of this paper is a theoretical. The introduced half-space reporting data structure used in this paper will take additional $O(n \log n)$ space [12] . To have an intuitive understanding of the level of such additional memory usage, we would point out that the weights of neural networks takes $O(n d)$ space. So when the neural network is over-parameterized, i.e., $d$ is very large, the additional memory usage by our algorithm is not the dominant term in space complexity.

Reviewer comment (Writing):
On Page 3, what is the definition for an $\epsilon$-net?

Author response:
The definition of $\epsilon$-net is defined as follows: Given a metric space $(X, d)$, a subset $Y$ of $X$ is said to be an  $\epsilon$-net if

    (i) For $a, b \in Y$ , we have $d(a, b) \ge \epsilon$, and

    (ii) For all $x \in X$, there exists an $a \in Y$ such that $d(x, a) < \epsilon$.

Reviewer comment (Theory):
It seems only optimal adversaries are analyzed (e.g., in theorem 4.4), but most cases it is impractical to have an optimal adversary.

Author response:
We would point out that if the adversarial example construction is not optimal, it does not hurt the acceleration achieved by our algorithm. However, we also admit that if we are not using the optimal adversarial construction, then the error bound in Theorem 4.4 for weights will be worse. Depending on what specific assumptions we make for the non-optimal adversarial examples, e.g. we are using adversarial construction $A$ instead of the optimal one $A^*$, the error bound for $W$ will be correspondingly worse.

Reviewer comment (Experiments):
Could authors provide some discussions on and thoughts on if the activation sparsity could generalize to deeper nets?

Author response:
As mentioned in point 1, some empirical works observe a similar phenomenon of sparsity in multi-layer neural networks. Hence the intuition might still be avoiding making redundant $0$-update for non-activated neurons. The major difficulty would then be designing new data structures that could efficiently maintain activated neurons during the adversarial training procedure, which we make as a very interesting future direction.

Reviewer comment (Theory):
Is there any assumption on the adversarial example oracle A in Algorithm 1? I don't seem be able to find any reference to its complexity analysis in the main paper. Usually adversarial examples are found by iterative projected gradient descent (on the input x) and their runtime should also be taken into account.

Author response:
Thank you for your valuable questions. We would revise the typos in the next version.  
1. We take standard adversarial example construction in our algorithm. So if we let $T$ denote the complexity for the adversarial example oracle, then the improvement our algorithm achieved is $o(nmdT)$ compared to standard algorithm $\Omega(nmdT)$.