Paper: Mitigating the Curse of Dimensionality for Certified Robustness via Dual Randomized Smoothing (ICLR-2024)

Reviewer comment (Theory):
This paper does not convince me that DRS will be better at RS in terms of classification accuracy. Although experiments can partially validate this idea, I think some form of theoretical analysis is needed in order to explain that DRS is at least not a lot worse than RS, because experiments can be easily tuned to show better results.

Author response:
The classification accuracy of DRS compared to that of RS is data-dependent. In the case of high-dimensional, real-world datasets, particularly those with significant information redundancy like images, the classification accuracy of DRS can be very close to that of RS. This comes from a well-known fact that images only occupy a very small subspace of the image space spanned by pixels, especially for high-resolution images. Hence, each sub-sample obtained through down-sampling retains a substantial portion of the original information. [1] claims that masking a high proportion of the input image, e.g., 75%, yields a meaningful self-supervisory task. In general, the classification accuracy of DRS will not be a lot worse than RS for those data with high information redundancy if the data is properly partitioned. 

To verify that, we conducted a comparative experiment to evaluate the classification accuracy on the CIFAR-10 dataset using $f$, $f^l$,$f^r$, and $f^l+f^r$. For all classifiers, we employed a standard training strategy; we used the ResNet110 as the classification model, and SGD as the optimizer with  $learningrate =0.1,~ momentum=0.9,~ and~weightdecay=5*10^{-4}$. We used the standard augmentation including random crop and flip. Each model was trained for 100 epochs with a learning rate decay implemented via $CosineAnnealingLR$. The experiment result is shown in the following table. The result reveals that DRS shows a very similar accuracy performance to RS for this image dataset.  

| Model     | Classification Accuracy (%)             |
| --------- | -------------------- |
| $f$      | $94.0~(\pm0.1)$ |
| $f^l+f^r$ | $93.9~(\pm0.2)$ |
| $f^r$    | $92.8~(\pm0.1)$ |
| $f^l$     | $92.8~(\pm0.2)$ |

For the improvement in the certified robustness of DRS, we primarily attribute it to its ability to guarantee the same robustness as RS using a lower noise variance. In Tables 3 and 4, we show that under consistency training, DRS under a lower noise variance achieves a higher certified accuracy and upholds a comparable level of certified robustness as RS, but encounters some accuracy drop when the noise variance increases to match that of RS.

Reviewer comment (Theory):
If partitioning into 2 spaces is good for mitigating curse of dimensionality, what about 3 spaces and even more? Theoretically speaking they should mitigate the problem better, but why choose 2?

Author response:
Further partitioning the input is quite promising for enhancing the robustness upper bound. Reflecting on your discerning insight, in Appendix A.3 of the revised manuscript, we further explore the feasibility of k-partitioning based smoothing. By deriving a general architecture of the smoothed classifier $g$ using $k$-partitioning, in Equation (26), we reformulate the guarantee of certified robustness as an optimization problem. Our findings in Appendix A.3 indicate that:

1. for $k=2$,  this optimization is proven to be convex, where the global minimum to guarantee the certified robustness is achieved by $\mathop{p'}\nolimits_A^1=\mathop{p'}\nolimits_A^2=\frac{1}{2}\tilde p$. This is in alignment with our assertion in Equation (11). 
2. For $k>2$, this optimization is non-convex. Providing a numerically stable solution for the global minimum under these circumstances is challenging (the minimum might not exist). A saddle point is achieved when $\mathop{p'}\nolimits_A^j  = {{\tilde p} \mathord{\left/{\vphantom {{\tilde p} k}} \right.} k} , \forall j \in \{1, \ldots, k\}$, but it does not strictly ensure the certified robustness. 

We present an in-depth analysis and an illustrative visualization of this $k$-partitioning based smoothing in Appendix A.3, which may help future research endeavors.

Reviewer comment (Writing):
First word in Section 5.3 is missing capitalized first letter.

Author response:
In our revised version, we corrected this typo and added an explanation of the down-sampling process: "Considering the high information redundancy of adjacent pixels in the image, we utilize two $2\times2$ pixel-index matrices as the down-sampling kernels. Each kernel selectively retains pixels along one of the two diagonals within the $2\times2$ image block, as depicted in Figure 2. The two sub-images are generated by sliding the two kernels across the original image, moving with a stride of 2.".

Reviewer comment (Writing):
A bit more explanation would be useful on how the down-sampling is performed to break each input into its two parts. Specifically, it is sort of glossed over in text and left to the reader to interpret the procedure based on Figure 2. A simple one to two sentence explanation in the main body of the paper would suffice, preferably with an explicit pointer to the graphic of Figure 2, where the authors may want to state that the index set corresponds to a two-part partition of the images’ pixel indices, and \(x^l\) is the vector composed of \(x\)'s pixel values \(x_i\) with \(i\in\text{idx}\), and \(x^r\) is the vector composed of \(x\)'s pixel values \(x_i\) with \(i\notin\text{idx}\).

Author response:
In our revised version, we corrected this typo and added an explanation of the down-sampling process: "Considering the high information redundancy of adjacent pixels in the image, we utilize two $2\times2$ pixel-index matrices as the down-sampling kernels. Each kernel selectively retains pixels along one of the two diagonals within the $2\times2$ image block, as depicted in Figure 2. The two sub-images are generated by sliding the two kernels across the original image, moving with a stride of 2.".

Reviewer comment (Reproducibility):
Where does the \(5*10^{-7}\) come from in (4)? Is this from a pre-specified sample size in RS? It would be nice to briefly mention this in Section 3.2.

Author response:
In our revised manuscript, we added an explanation that "the parameter $5 \times 10^{-7}$ is derived from a predefined error margin ($1 \times 10^{-6}$) between the estimated probability and true probability over the origin-symmetric distribution.". Indeed, this parameter maintains a relationship with the sample size.

Reviewer comment (Theory):
Why restrict the two low-dimensional smoothing procedures to share the same variance? How restrictive/suboptimal is this in practice? Will your guarantees generalize to the setting with different smoothing variances?

Author response:
1. Yes, DRS guarantees generalization to the setting with difference smoothing variance. Following your insights, we analyze the generalizability of DRS with different smoothing variances in Appendix A.4 of the revised manuscript. Let $\eta$ denote the ratio of standard deviations selected for two sub-inputs, specifically $\eta={{\sigma} ^r}/{{\sigma} ^l}$. Our finding in Appendix A.4 is that the calculation of the worst-case robustness provided by DRS with different smoothing variances can be formulated into a convex optimization problem. A global minimum that guarantees the certified robustness of DRS is achieved by making the first-order derivative equal to zero, which is: 
   $$\eta *\frac{d}{{{d{\mathop{p'}\nolimits_A^l}}}}{\Phi ^{ - 1}}\left( {\mathop{p'}\nolimits_A^l + 1 - \tilde p} \right) = \frac{d}{{{d{\mathop{p'}\nolimits_A^l}}}}{\Phi ^{ - 1}}\left( \mathop{p'}\nolimits_A^l \right).$$ 
   For $\eta=1$, the solution of the above equation is $\mathop{p'}\nolimits_A^l=\mathop{p'}\nolimits_A^r=\frac{1}{2}\tilde p$, which is in alignment with our assertion in Equation (11). An illustrative visualization of this problem is presented in Figure 5. Please kindly refer to the updated manuscript for more details. 
2. Yes, the reviewer is correct that restricting the uniform variance for the two sub-images is suboptimal in practice. Nonetheless,  the substantial information overlap in data conveyed by the sub-images indicates that their respective optimal variance levels are close. Inspired by the input-dependent RS [2], determining the optimal variance for each image within the DRS framework is promising work.

Reviewer comment (Theory):
The authors claim that their bound is tight. Perhaps I missed this, but where is this claim proven/justified?

Author response:
The tightness of the certified robustness of RS is proven in [3]. For DRS, we assume that the certified radius directly derived from RS is tight. Given that ${p_B} =  1 - {p_A}$,  the inequalities presented in Equations (7), (8), (9), and (11) are transformed into equalities. This indicates the smallest of ${\left\| {{{\delta} ^l}} \right\|_2} + {\left\| {{{\delta} ^r}} \right\|_2}$ to break DRS is $\sigma \left( {{\Phi ^{ - 1}}\left( {p_A^l} \right) + {\Phi ^{ - 1}}\left( {p_A^r} \right)}\right)$. Consequently, in the worst case situation that ${\left\| {{{\delta} ^l}} \right\|_2} = {\left\| {{{\delta} ^r}} \right\|_2}$, the largest certified robustness provided by DRS is $R=\frac{\sigma }{{\sqrt 2 }}\left( {{\Phi ^{ - 1}}\left( { {p_A^l} } \right) + {\Phi ^{ - 1}}\left( {{p_A^r} } \right)} \right)$, which proves the tightness of DRS robustness bound. We added this justification in our revised manuscript in Appendix 5.

Reviewer comment (Novelty):
The improvement from $O(1/\\sqrt{d})$ to $O(1/\\sqrt{m}+1/\\sqrt{n})$ is mostly a constant change.

Author response:
For $m=n=\frac{d}{2}$ implemented in DRS, the improvement of the upper bound is a constant factor of $\sqrt 2$. Reflecting on your discerning insight, in Appendix A.3 of the revised manuscript, we further explore the feasibility of k-partitioning based smoothing. By deriving a general architecture of the smoothed classifier $g$ using $k$-partitioning, in Equation (26), we reformulate the guarantee of certified robustness as an optimization problem. Our findings in Appendix A.3 indicate that:

1. for $k=2$,  this optimization is proven to be convex, where the global minimum to guarantee the certified robustness is achieved by $\mathop{p'}\nolimits_A^1=\mathop{p'}\nolimits_A^2=\frac{1}{2}\tilde p$. This is in alignment with our assertion in Equation (11). 
2. For $k>2$, this optimization is non-convex. Providing a numerically stable solution for the global minimum under these circumstances is challenging (the minimum might not exist). A saddle point is achieved when $\mathop{p'}\nolimits_A^j  = {{\tilde p} \mathord{\left/{\vphantom {{\tilde p} k}} \right.} k} , \forall j \in \{1, \ldots, k\}$, but it does not strictly ensure the certified robustness. 

We present an in-depth analysis and an illustrative visualization of this $k$-partitioning based smoothing in Appendix A.3, which may help future research endeavors.

Reviewer comment (Reproducibility):
Part of DRS in Section 4.2 is a bit unclear. Specifically, how is the downsampling function implemented, and how does it make sure that every subsample gets enough spatial info from the original input?

Author response:
In our revised paper, we added the explanation that: "Considering the high information redundancy of adjacent pixels in the image, we utilize two $2\times2$ pixel-index matrices as the down-sampling kernels. Each kernel selectively retains pixels along one of the two diagonals within the $2\times2$ image block, as depicted in Figure 2. The two sub-images are generated by sliding the two kernels across the original image, moving with a stride of 2."

Reviewer comment (Theory):
(Based on first point in weakness) Have you considered further partitioning the input, for instance, into $k$ subsamples, and then leveraging $k$ randomized smoothing classifiers for the final decision? Would such an approach potentially enhance the dimension dependence even more, perhaps improving the robustness radius from $O(1/\sqrt{d})$ to something like $O(\sum_{i=1}^k 1/\sqrt{d_k})$, where $\sum_{i=1}^k d_k = d$?

Author response:
For $m=n=\frac{d}{2}$ implemented in DRS, the improvement of the upper bound is a constant factor of $\sqrt 2$. Reflecting on your discerning insight, in Appendix A.3 of the revised manuscript, we further explore the feasibility of k-partitioning based smoothing. By deriving a general architecture of the smoothed classifier $g$ using $k$-partitioning, in Equation (26), we reformulate the guarantee of certified robustness as an optimization problem. Our findings in Appendix A.3 indicate that:

1. for $k=2$,  this optimization is proven to be convex, where the global minimum to guarantee the certified robustness is achieved by $\mathop{p'}\nolimits_A^1=\mathop{p'}\nolimits_A^2=\frac{1}{2}\tilde p$. This is in alignment with our assertion in Equation (11). 
2. For $k>2$, this optimization is non-convex. Providing a numerically stable solution for the global minimum under these circumstances is challenging (the minimum might not exist). A saddle point is achieved when $\mathop{p'}\nolimits_A^j  = {{\tilde p} \mathord{\left/{\vphantom {{\tilde p} k}} \right.} k} , \forall j \in \{1, \ldots, k\}$, but it does not strictly ensure the certified robustness. 

We present an in-depth analysis and an illustrative visualization of this $k$-partitioning based smoothing in Appendix A.3, which may help future research endeavors.