Paper: The Lipschitz-Variance-Margin Tradeoff for Enhanced Randomized Smoothing (ICLR-2024)

Reviewer comment (Writing):
The paper is at times verbose, and at times unclear. Some comments are exaggerated or incorrect (See below).

Author response:
We have revised the paper in an effort to make it clearer.

Reviewer comment (Novelty):
In my view, the contributions of employing a better concentration inequality to provide tighter tail guarantees for finite samples is small, as is the fact of changing the function that maps the sub-classifier predictions to the probability simplex.

Author response:
Our approach generalizes the usage of argmax as a simplex map and provides insight into why methods like (Salman et al. 2019) which reduce Lipschitz of $f$ are crucial.

Reviewer comment (Theory):
Throughout the paper, the authors refer to softmax (and other functions) as "projections" onto the probability simplex. The softmax is not a projection onto the simplex. In fact, this reviewer believes that the softmax is not a projection onto any set (is it?). Throughout the manuscript, the word "projection" is used in lieu of the term "function" or "map".

Author response:
We have modified the notation and replaced "simplex projection" by "simplex map".

Reviewer comment (Theory):
Randomized smoothing can provide a certified radius at varying levels, depending on the strength of the added noise. It is clear to me that leveraging the fact that \( f \) is Lipschitz continuous allows the authors to find an optimal randomized smoothing strength, as indicated in Prop. 4. Such a smoothing strength results in a certified radius. What is unclear to me is, what can be done if one requires certification at a different radius?

Author response:
As we are doing Randomized Smoothing on an already Lipschitz classifier, we can as in Randomized Smoothing alone change the magnitude $\sigma$. It will not be optimal: the cross effect of Randomized Smoothing and Lipschtz Continuity will not be maximal, but it allows to reach for a potentially greater certified radius.  
We have also generalized our procedure for a simplex of mass $r$, changing $r$ allows to adapt to one targetted certified radius, as discussed in Section 3.4.

Reviewer comment (Theory):
Page 3: The definition of \( L(f) \) that the authors use requires \( f \) to be differentiable. This assumption is not necessary (as best as I can tell) for their results to hold (and indeed, not applicable to ReLU networks). So, just define \( L(f) \) by its standard definition (without assuming differentiability).

Author response:
We have modified the definition of $L(f)$ by:  

$L(f) = \sup_{\substack{x, x' \in \mathcal{X} \\ x \neq x'}} \frac{\lVert f(x) - f(x') \rVert_2}{\lVert x - x' \rVert_2} \enspace$.

Reviewer comment (Theory):
The authors comment on some related works pertaining to the analysis of Lipschitz-continuity for devising certified radii. There exist extensions of these ideas that refine the analysis to requiring only local Lipschitzness, as in [A], which lead to improvements on the certified radius. Can the authors comment on relations to that work, and on whether their results could apply in a local way as in [A]?

Author response:
We have devised a Theorem to bound $||\nabla \Phi^{-1} \circ \tilde{f}_k (x) ||$. This theorem can be used to derive bound on a the local Lipschitz constant.

Reviewer comment (Writing):
Page 4: The authors mention that "these bounds are relevant for the composite classifier defined as \( \Phi^{-1} \circ f \)". However, I believe they mean the classifier \( \tilde{F}_s \), no?

Author response:
We have corrected the error, we meant the classifier $\Phi^{-1} \circ \tilde{f}_k$.

Reviewer comment (Theory):
The authors argue in Section 3.3 that different functions \( s(z) \) have different impacts in terms of the variance of \( s(Z) \) vs the variance in \( Z \). This is quite obvious, and, respectfully, I find the Example 1 not useful/trivial. It is unclear, however, how such a map should be chosen to be optimal. It seems to me (from their experimental section) that they simply compute different options based on different alternatives for \( s(z) \) and choose the one that results in larger empirical margin. Is this correct? Shouldn't one be able to provide a "best" choice for \( s(z) \)?

Author response:
For future works, I think, it is not obvious that there is a consensus. (60 \% is $\mathrm{argmax}$ rest $\mathrm{sparsemax}$).  
We are going to modify the procedure and add a phase where we determine $s$ and $t$ on a first $n_0$ sample of $\delta_i$, in the same manner as \cite{cohen_certified_2019}.  
Then run the standard, RS procedure with the classifier $F_{s, t}$ on another sample of size $n$.  
This new procedure is statistically correct.

Reviewer comment (Writing):
After Prop. 3, the authors mention that "This refinement on the bound was possible by supposing Lipschitz continuity on the base classifier \( f \). Note that its Lipschitz constant can be arbitrarily high, so this assumption is quite light". What do the authors mean by "light" here?

Author response:
It means that the Lipschitz constant $L(f)$ can be arbitrarily high at the end we come up with a bound which does not depend on  $L(f)$. So it is not a restrictive hypothesis to assume and this conducts to a better bound for most networks used in Randomized Smoothing.

Reviewer comment (Theory):
Page 1: The authors write that there's a "circular dependency: the regularity of the smoothed classifier depends on the Lipschitz property of the base classifier and the variance of the Gaussian convolution which governs the induced level of smoothness." Yet, I don't see how this is a circular dependency.

Author response:
We have removed this formulation. It was used because the Lipschitz constant of the base classifier plays a dual role in the MC variance and the Lipschitz constant of the smoothed classifier.

Reviewer comment (Writing):
Page 3: What do the authors mean by "furnished by \( f \)"?

Author response:
We reformulated this sentence by "given by $f$".

Reviewer comment (Writing):
Page 3: "concerning the \( \ell_2 \) norm" → "w.r.t...."

Author response:
Same as above.

Reviewer comment (Writing):
Page 3: In defining the margin of \( f \) at \( x \), it should be "of \( f \) at \( (x,y) \)", since its definition uses the correct label \( y \).

Author response:
We modified the definition of the margin accordingly.

Reviewer comment (Writing):
Prop. 1: The first sentence is incomplete. Please revise English usage.

Author response:
We rewrote the sentence.

Reviewer comment (Writing):
The authors use the term "sub-classifier" to refer to the un-robustified function \( f \), alas sometimes they refer to it as a "classifier" (as in Prop. 3).

Author response:
We corrected it.

Reviewer comment (Writing):
It might help the reader to make explicit what the final classifier is (which I believe is \( \arg\max_{s,t} \tilde{F}^t_s(x) \), no?).

Author response:
We corrected it and defined it clearly in the paper now.

Reviewer comment (Writing):
In the conclusion: "In this paper, we demonstrate a significant correlation between the variance of randomized smoothing and two critical properties of the base classifier." I don't think the authors demonstrate any "correlation" between these things. Maybe they meant "connection"?

Author response:
We corrected this term usage.

Reviewer comment (Writing):
Most of the randomized smoothing approaches simplify the margin $-\max_{y\neq k}\hat f_k(x)\geq -(1-\hat f_y(x))$ (Cohen et al., 2019). Can you clarify where in the paper of Cohen? Are you referencing their choice $p_A=1-p_B$ in top of their p6?

Author response:
Yes, it was the message we wanted to convey. In (Cohen et al.2019) they reduce the multi-class setting to a binary class setting, this reduction comes at the expense of a degraded certified radius.
We have rewritten this part to make it clearer, please see Appendix C.

Reviewer comment (Evaluation):
For each combination of (s, t), we determine the risk-corrected margin $M_2(p_{st} , \alpha)$. Is it statistically correct (w.r.t $\alpha$)? Since you perform multiple tests with the same scores $f(x+\delta_i)$?

Author response:
We have modified the procedure and added a phase where we determine $s^*$ and $t^*$ on a first $n_0$ sample of $\delta_i$, in the same manner as \cite{cohen_certified_2019}.
Then run the standard, RS procedure with the classifier $F_{s^*, t^*}$ on another sample of size $n$.

Reviewer comment (Writing):
It is not always clear from context what the definitions of functions are. For example: $\hat f$ is implicitly defined in top of page 4, I assumed you meant $\hat f_k()$ in the argmax, so $\hat f:\mathcal{X}\rightarrow\mathbb{R}^c$.

Author response:
We have modified this section and add a proper definition of the smooth sub classifier $\tilde{f}$.

Reviewer comment (Writing):
$\Phi$ is a scalar function (per its definition), so what is the meaning of $\Phi^{-1}\circ \hat f$? Is it vectorized?

Author response:
We have modified the notation in the paper and only note $\Phi^{-1}(\tilde{f}_k(x))$ with $\Phi^{-1} : [0, 1] \mapsto \mathbb{R}$.

Reviewer comment (Writing):
Is $g$ a scalar or a vectorized function in Eq (3)?

Author response:
We have modified the notation in the paper and only note $\Phi^{-1}(\tilde{f}_k(x))$ with $\Phi^{-1} : [0, 1] \mapsto \mathbb{R}$.

Reviewer comment (Theory):
In Sec 2.1 the notation $L()$ is defined as a maximum of gradient norm over $\mathbb{R}^d$. But the gradient is only defined for scalar functions, not for multivariate functions (in which case it is a Jacobian, not a gradient). This makes sense to define $l(g)=\max_k L(f_k)$. But, then, if $g$ is multivariate, how can you write $L(g)$? What does it mean?

Author response:
We have modified the definition of the Lipschitz constant as $L(f) = \sup_{\substack{x, x' \in \mathcal{X} \\ x \neq x'}} \frac{\lVert f(x) - f(x') \rVert_2}{\lVert x - x' \rVert_2} \enspace$.
We also remove the notation $l(f)$ and will use $L(f_k)$ instead.

Reviewer comment (Writing):
I suggest to define clearly each notation, and specify the domain of each function, not in an implicit manner in the middle of a formula.

Author response:
We have done accordingly in the paper.

Reviewer comment (Experiments):
What is the runtime cost of your method compared to randomized smoothing?

Author response:
We have integrated runtime cost in the tables.

Reviewer comment (Writing):
The paper can be hard to read. In p6: "Cohen et al. (2019); Salman et al. (2019); Carlini et al. (2023) use argmax simplex projection they can use the Clopper-Pearson Bernouilli tailored confidence interval ... etc" — This paragraph should be rewritten, the first two sentences are not grammatically correct.

Author response:
We have integrated your remarks.

Reviewer comment (Writing):
In the abstract, it is unclear what “variance introduced by RS”, “simplex projection”, and “variance-margin tradeoff” mean. It is also unclear what Bernstein’s inequality has to do with any of this.

Author response:
**Abstract: It is unclear what “variance introduced by RS”, “simplex projection”, “variance-margin tradeoff” mean. It is unclear what does Bernstein’s inequality have to do with any of this.**  
We have modified the abstract to make these notions more clear.

Reviewer comment (Writing):
In the introduction, it would be good to define prediction margin. All methods with certified radius deal with a way of controlling/estimating the Lipschitz constant of the network, so it is unclear why Tsuzuku et al. is specifically mentioned. The statements are also wordy (“radius that encompasses … constant”), and could be simplified, for example using $|f(x) - f(x + \delta)| \leq L ||\delta||$.

Author response:
**Introduction: Would be good to define prediction margin. All methods with certified radius deal with a way of controlling /estimating the Lipschitz constant of the network, it is unclear why is Tsuzuku et. al. specifically mentioned. The statements are also quite wordy here (“radius that encompasses … constant”), a simple $| f(x) - f(x + \delta) | \leq L || \delta ||$ would suffice here.**  
We have included the definition of margin and Lipschitz in the introduction.

Reviewer comment (Presentation):
In Figure 1, the meaning of radius binding, risk shift, and simplex projection is unclear. The term “Risk” is used throughout the introduction without a definition.

Author response:
**Figure 1: What is radius binding, risk shift, simplex projection? “Risk” is used throughout the introduction without a definition.**  
We have added a sentence in the introduction to define the risk: the probability that the MC classifier outputs the wrong answer. We have added a definition of the risk $\alpha$ in the related work section on randomized smoothing.
We also have defined what is a simplex map (previously called simplex projection) in the introduction.

Reviewer comment (Writing):
On page 2, the circular dependency mentioned in paragraph 2 is unclear — what exactly is the circular dependency?

Author response:
**The circular dependency mentioned in paragraph 2 was not clear, what exactly is the circular dependency?**  
We have removed the use of this term. It was used to describe the different impacts of the Lipschitz constant of the base classifier on the MC variance and on the Lipschitz constant of the base classifier.

Reviewer comment (Writing):
The definition of R(f, x) is written in a convoluted fashion. Isn’t it simply $\min_\epsilon$ such that $\exists$ adversarial example $\in B(x, \epsilon)$? The benefit of the written form of the definition is unclear.

Author response:
**The definition of R(f, x) is written in a very convoluted fashion, isn’t it simply $\min_\epsilon$ such that $\exists$ adversarial example $\in B(x, \epsilon)$. The benefit of the written form of the definition is unclear.**  
We have modified the definition of $R(f, x)$ accordingly.

Reviewer comment (Novelty):
The correct earliest reference for Randomized Smoothing is not Cohen et al. 2019. The related work section in Cohen et al. 2019 should be consulted for correct references.

Author response:
**The correct earliest reference for Randomized Smoothing is not Cohen et al. 2019, please see the related work section in Cohen et al. 2019 for the correct references.**  
We have added the previous reference of Randomized Smoothing, Lecuyer 2019, Li 2019.

Reviewer comment (Theory):
Defining the certified radius for the composition of $\Phi^{-1}$ and $\tilde f$ is confusing, since the radius is to be used for certifying $\tilde f$, and not the composition.

Author response:
**Defining the certified radius for the composition of and is confusing, since the radius is to be used for certifying $\Phi^{-1}$, and $\tilde{f}$, not the composition.**  
When performing Randomized Smoothing we estimate the smoothed classifier $\tilde{f}: R^d \mapsto [0, 1]$.
With this estimation, we have by composing with $\Phi^{-1}$ the classifier $\Phi^{-1} \circ \tilde{f} : R^d \mapsto R$ which enjoys a better certified radius.

Reviewer comment (Writing):
Section 2.4 (Margins and Robustness) is too wordy (“promise in bolstering classifier defense”, “inherent synergy between them”, “our exploration dives deep into this nexus unravelling novel insights …”). The main message — greater margin = larger certified radius (at fixed L) — could be stated in a single line.

Author response:
**Section 2.4: Margins and Robustness — This is again too wordy “promise in bolstering classifier defense”, “inherent synergy between them”, “our exploration dives deep into this nexus unraveling novel insights …”. The message again simply seems to be that greater margin = larger certified radius (at fixed L), and could be stated in a single line.**  
We have reformulated those sentences to make them clearer, please see section 2.4.

Reviewer comment (Presentation):
In Section 3.1 (Relation between certified Radii), while the highlight is the last paragraph about variance playing a crucial role in RS certificates, the contribution of the rest of the section to this message is unclear. It may be better to stress this strong point and detail Table 3.1.

Author response:
**Section 3.1: Relation between certified Radii. The highlight of this section seems to be the last paragraph, that variance plays a crucial role in the RS certificates. It is unclear what does the rest of the section adds to this message. Perhaps it would be better to simply stress this strong point, and detail Table 3.1.**  
We have put this section in the Appendix as it does not contain important insights for the rest of the contribution.

Reviewer comment (Presentation):
Table 3.1 is hard to parse. Simply plotting the difference between R2 and R3 might convey the message more clearly.

Author response:
**Table 3.1 is hard to parse, could simply plotting the difference between R2 and R3 convey the message better?**  
We have added a column with the difference between the radius.

Reviewer comment (Theory):
On page 3, Proposition 1, it seems the condition for correct prediction should be margin > 2 * Lipschitz Constant * perturbation radius, yet the paper uses $\sqrt{2}$ instead of 2. This discrepancy needs clarification.

Author response:
**Proposition 1**. Isn’t the condition for correct prediction simply that margin > 2 * Lipschitz Constant * perturbation radius? ( Assume class 1 is majority at x, class 2 is second highest, then $|f_1(x) - f_1(x + v)| \leq L ||v|| ) =L \epsilon$, and $|f_2(x) - f_2(x+v)| \leq L \epsilon$. Then, if $f_1(x) - f_2(x) > 2 L \epsilon$, then $f_1(x+v) > f_2(x +v)$. Why is the constant $\sqrt{2}$, instead of 2?

The demonstration that you gave to certify $x$ for classifier $f$ is true when you assume $f_k$ to be $L$ Lipschitz continuous for every $k$. Here we have the assumption that $f$ is $L$ Lipschitz continuous, hence we can obtain a stronger certificate.

Tsuzuku et al proposed a tighter version of this bound in Proposition 1 of [1]. 

[1] Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks

Reviewer comment (Theory):
In Appendix page 12, Proof of Proposition 3, the definition of $u \in R^c$ is unexplained and unused.

Author response:
###  Appendix P12, Proof of Proposition 3.

### Why is $u \in \mathbb{R}^d$ defined? Doesn’t seem to be used in the proof

It is a typo and has been corrected. It is used in the next proof for the bound on $L(\tilde{h}_k)$, but for $L(\tilde{h})$ it is not required.

Reviewer comment (Theory):
**Major 1** in Appendix page 12, Proof of Proposition 3: The step $v^T \nabla \tilde h(x) = (1/\sigma^2) E[…]$ is unclear. Expanding $v^T \nabla \tilde h(x)$ gives $v^T \nabla \tilde h(x) = E[v^T \nabla h(x + \delta)]$, but the transition from here is not explained.

Author response:
### Major 1 Why is $v^\top \nabla \tilde{h} (x) = \frac{1}{\sigma^2}E[...]$ Expanding $v^T \nabla \tilde{h} (x)$ gives $E[v^T \nabla h(x + \delta)]$ it is unclear how to proceed from here : 

We forgot to include Stein's Lemma [Charles M. Stein. Estimation of the mean of a multivariate normal distribution. The Annals of
Statistics, 9(6):1135–1151, 1981] in the proof. It has been added.

To compute $\nabla \tilde{h} (x)$ we use Stein's Lemma :

Lemma:  (Stein's lemma) \\
Let $\sigma > 0$, let $h : R^d \mapsto R$ be measurable, and let 
$\tilde{h}(x) = E_{\delta\sim {\cal N}(0, \sigma^2 I)}[h(x + \delta)]$. Then $\tilde{h}$ is differentiable, and moreover,
$$ 
\nabla \tilde{h}(x) = \frac{1}{\sigma^2} E_{\delta\sim {\cal N}(0, \sigma^2 I)}[ \delta h(x + \delta)] \ .
$$   

We will add it to the proof.

Reviewer comment (Theory):
In the step marked (i) in Appendix page 12, Proof of Proposition 3, $|h(x + \delta) - h(x + \delta’)|$ is replaced by $\min(1, 2L) |v^\top \delta|$. The justification for this replacement is unclear — shouldn’t the bound be $L ||\delta - \delta'||$?

Author response:
### In the step marked (i) in the proof, is there some typo? $|h(x+\delta) - h(x+\delta^\prime)|$ is replaced by $\min(1, 2 L)|v^\top \delta|$}

It is explained later in the proof :

"where $(i)$ follows from the Lipschitz assumption on $h$ and the fact that $\|\delta-\delta'\| = \| 2\tilde{\delta} \| = \|  2(v^T\delta)v\| = 2|v^T\delta|$ and that $|h(x+\delta) - h(x+\delta')| \leq 1$"


Using that $|h(x+\delta) - h(x+\delta^\prime)| \leq l(h) \|\delta-\delta'\|$ and what is mentioned above, we get $|h(x+\delta) - h(x+\delta^\prime)| \leq 2 l(h) |v^T\delta|$ and $|h(x+\delta) - h(x+\delta^\prime)| \leq 1$ as $h(x) \in [0, 1]$. Finally, it is inferior to the minimum of the two bounds: $|h(x+\delta) - h(x+\delta^\prime)| \leq \min(1, 2 l(h))|v^\top \delta|$.

Reviewer comment (Theory):
It is unclear how equation (4) is obtained using the bound in the preceding paragraph. Additional explanation would help readers unfamiliar with the proof in Cohen et al. 2019.

Author response:
### It is unclear how one obtains (4) by using the bound mentioned in the paragraph preceding it. A few lines would be great for readers not familiar with the proof in Cohen et al. 2019.
This is a typo and has been corrected

Reviewer comment (Novelty):
The Gaussian Poincaré inequality presented seems very similar to other well-known concentration inequalities for functions of Gaussian random variables. Contextualization is needed, for example comparing it to the standard concentration inequality for Lipschitz functions of sub-Gaussian random variables: $P(|h(Z) - E h(Z)| > \delta) \leq \exp(-c \delta^2 / L(h))$ (see Theorem 5.2.2 in High Dimensional Probability by Roman Vershynin).

Author response:
### The Gaussian Poincare inequality seems to be very similar to other well known concentration inequalities for functions of gaussian random variables $P(|h(Z) - Eh(Z)| > \delta ) \leq \exp(-c \delta^2 / L(h))$. A brief contextualization would be great. For instance, how does this relate to the standard concentration inequality for Lipschitz functions of sub gaussian random variables: (for a reference, see Theorem 5.2.2 in High Dimensional Probability by Roman Vershynin)

A discussion has been added to the paper in Section 3.2.
The sub-Gaussian inequality involving the Lipschtiz constant of $s_k \circ f$, 
For $\delta_0, \delta_1, \dots, \delta_n$ i.i.d standard normal, for all $\mathrm{shift} > 0$:
$$    \mathbb{P}[ \mathbb{E}[s_k(f(X + \delta_0))] - \frac{1}{n} \sum_{i=1}^n s_k(f(x + \delta_i)) \geq \mathrm{shift}] \leq \exp{\left( -\frac{\mathrm{shift}^2}{2 L(s_k \circ f)^2}\right)}, 
$$
The issue here is that computing $L(s_k \circ f)$ is NP-hard for common neural networks and the Lipschitz constant as a bound can overestimate the actual empirical variance.
Most of the time the empirical variance provides a tighter bound on the true variance than $\sigma^2 L(s_k \circ f)^2$, which does not include the locality of the input $x$.

Reviewer comment (Theory):
In equation (5), the right-hand side does not show where $\alpha$ appears. It is unclear how $p_1, p_2$ depend explicitly on $\alpha$. The likely dependence is through the number of samples n affecting both $\alpha$ and $p_1, p_2$, but this is not obvious from (5).

Author response:
### Eq (5), where is alpha in the  RHS? Specifically, it is unclear how the  $p_1, p_2$ have any explicit dependence on $\alpha$. I believe the dependence is that the number of samples n affects both alpha and $p_1, p_2$. But this is not clear at all by looking at (5).

In our framework the risk $\alpha$ is taken as a constant and for different sampling numbers it is the risk shift that is going to vary.
We have made clear and apparent the dependency of $\alpha$ on $\bar{p}_1$ and $\bar{p}_2$. And on the risk corrected certified radius $R_2(\bar{p})$.

Reviewer comment (Reproducibility):
In Algorithm 1, line 4, the nature of the simplex projections is unclear. The text mentions “for various projections … spanning softmax and sparsemax”, but which projections specifically are used?

Author response:
### What are the simplex projections being used in line 4, algorithm 1? The text mentions “for various projections … spanning softmax and sparsemax” — Which projections specifically?
Projection are ranging $s \in \{ sparsemax, softmax, argmax \}$ and also ranging accross different temperatures from [0.01, 50]. Details has been added in the Section 3.5 and Section 4.

Reviewer comment (Theory):
To compute margin $M_2$, it is assumed one needs the Lipschitz constant of the classifier $s$ composed with $f$. It is unclear how this constant is computed for each simplex projection chosen.

Author response:
### Now, in order to compute the margin $M_2$, I believe one needs to know the lipschitz constant of the classifier composed with, how is this computed for each simplex projection chosen?
To compute the certified radius of the smoothed classifier one needs indeed its Lipschitz constant. For an input $x$, and the smoothed classifier $\tilde{f}: \mathbb{R}^d \mapsto \mathbb{R}^c$ base on the sub classifier $f$, indeed, $\tilde{f}(x) := \mathbb{E}[s (f(x + \delta))]$. Let us call $p = \tilde{f}(x)$,  suppose that $p$ is sorted in decreasing order.
The certified radius is given by $R_2(p) = \frac{\sigma}{2} (\Phi^{-1}(p_1) - \Phi^{-1}(p_2) )$. Here the Lipschitz constant of $\Phi^{-1} \circ \tilde{f}$ is $1/\sigma$ as derived by (Salman et al, 2019).
There is no assumption needed on the Lipschitz constant of $s \circ f$. Therefore, for pure RS the choice of $s$ is not going to change the Lipschitz constant of $\tilde{f}$.

Reviewer comment (Reproducibility):
In the experiments, the method for constructing the 1-Lipschitz backbone is not explained, although it is relevant to the main contribution.

Author response:
### Experiments: How is the 1-Lipschitz backbone constructed? This should be detailed as it relates to the main contribution. 

We have taken the Small Sandwich architecture as described, in (Wang & Manchester, 2023). We will add a section in the appendix describing with more detail the architecture and the by-design Lipschitz layer.

Reviewer comment (Experiments):
In the experiments, the effects of changing the $\alpha$ calculation versus changing the certificate are unclear. Specifically, results are missing for: (a) using this paper’s $\alpha$ calculation on top of Cohen et al.’s method, and (b) using Clopper-Pearson $\alpha$ calculation on top of this paper’s certificate.

Author response:
### Experiments: The effects of the change in the $\alpha$ calculation vs the change in the certificate are unclear. In particular, what are the results when the $\alpha$ calculation in this paper is used on top of Cohen et. al.? Similarly, what are the results when the Clopper-Pearson $\alpha$ calculation is used on top of this paper’s certificate?

The Clopper-Pearson confidence interval can only be use for binomial samples, here we smooth scalar outputs. Therefore we can not directly compare the two.
We have included an ablation study to measure the impact of Bernstein's inequality over Hoeffding's inequality (previously used in (Lecuyer et al. 2019, Levine et al. 2020).

Reviewer comment (Theory):
There is a need to clarify and augment the paper with additional technical details to complete the proofs and correct the typos mentioned above.

Author response:
### (1) clarify and augment the paper with additional technical details to complete the proofs and correct the typos (details above),

We have added complementary details to the proof.

Reviewer comment (Theory):
The precise mathematical details of the certification procedure, from input to certificate, should be explained in detail, addressing issues noted in the weaknesses above.

Author response:
### (2) explain the precise mathematical details of the certification procedure from input to certificate (see details in weaknesses above), and

We have completed the algorithm describing the LVM-RS procedure, see Algo. 1.

Reviewer comment (Experiments):
An empirical ablation of the two components of the proposed approach should be conducted, as discussed in the weaknesses above.

Author response:
### (2) empirically ablate the two components of the proposed approach (see details above).

We have added Fig.4  to compare Hoeffding's inequality previously used in (Lecuyer et al,2019) and (Levine et al, 2020) with Bernstein's inequality and their impact on risk corrected margin.
We have not included the Clopper-Pearson interval as it can only be used on Bernoulli trials and not on scalar outputs.

Sparsemax, Argmax, and Softmax and their impact on margin are compared in Fig.5.

Reviewer comment (Presentation):
In terms of presentation, I feel the authors could add an overview paragraph for section 3. Sometimes I fail to connect each sub-sections. It may be good to map the flow of section 3 to Figure 1.

Author response:
**Weakness 1: In terms of presentation, I feel the authors could add an overview paragraph for section 3. Sometimes I fail to connect each sub-section. It may be good to map the flow of section 3 to Figure 1.**  
Section 3.1 has been removed from the body of the paper as it was not very insightful to the core message of the work.

Reviewer comment (Evaluation):
It will be good to discuss the computational cost of LVM-RS comparing to RS and deterministic Lipschitz.

Author response:
**Weakness 2: It will be good to discuss the computational cost of LVM-RS compared to RS and deterministic Lipschitz.**  
It has been added to the paper in the presentation of the LVM-RS procedure. Time computational costs have been added to the tables.

Reviewer comment (Theory):
Is there a typo in Eq 4 where \leq should be \geq? Currently it contradicts table 1.

Author response:
**Question 1: Is there a typo in Eq 4 where \leq should be \geq? Currently, it contradicts Table 1.**  
We corrected the typo in Eq. 4., it has been put in Appendix.

Reviewer comment (Reproducibility):
In algorithm 1, how many temperature do we need? And how to find those temperature?

Author response:
**Question 2: In algorithm 1, how many temperatures do we need? And how to find those temperatures?**  
In Algo. 1 we use 50 temperatures ranging from $0.1$ to $50$.  
Temperatures are chosen with a grid search. This has been added to the paper, in the description of the experiments.

Reviewer comment (Experiments):
How much computational cost that is induced by using multiple temperature? How sensitive the certified accuracy is with respect to the corrected margin?

Author response:
**Question 3: How much computational cost is induced by using multiple temperatures?**  
Using multiple temperatures is going to induce several application of simplex mappings, but the inference in the model and the sampling is done once. Hence it is not the bulk of the computation.  
We included the time of computation for RS, LVM-RS, and Lipschitz deterministic methods.  

**How sensitive the certified accuracy is with respect to the corrected margin?**  
Certified accuracy accounts for the frequency of input in which the certified radius is above a threshold $\epsilon$, the certified radius depends linearly on the margin.  
Can you please specify your question?