Paper: DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustness (ICLR-2024)

Reviewer comment (Experiments):
**No white-box evaluation.** The authors state that they compute attacks on the base model, thus framing them as white-box attacks (like Partial DOS; Shift, etc). However, these are evaluated as black-box transfer attack, and thus should be clarified on the paper.

Author response:
>>Thank you for pointing this out. We agree the current description might be confusing: Since the attacks require white-box access to the base models, they are indeed white-box. However, we agree they are in some sense transfer attacks since it is the base model rather than the DRSM itself being attacked. We have updated the **Section 7** for clarity. Let us know if you have further suggestions regarding presentations. We appreciate your feedback.

Reviewer comment (Experiments):
**Probable bad params for attacks.** The low success rates of attacks (especially GAMMA) might be due to a wrong initialisation. In the appendix, it is written that 200 as population size and query are used, but the number of queries for the GAMMA attack are computed as population_size * iterations. Also, the number of used sections is missing (which is a crucial point for the attack).

Author response:
>>We apologize for mentioning the query number as 200. We set the population size as 200, and ran it for 20 iterations. We have corrected it in the **Appendix A.4.8**. Also, for further confirmation, you can go to our provided code, and in the *line 68* of the `attack_malconv.py` file, you can find the initialization. It is -  
>>```  
>>attack = CGammaSectionsEvasionProblem(section_population, CEnd2EndWrapperPhi(net), population_size=200, penalty_regularizer=1e-12, iterations=20, threshold=0.5)  
>>```

Reviewer comment (Experiments):
**Dataset concerns.** While the release of a goodware dataset is for sure a great contribution, I am doubtful on the composition of such corpus. In particular, the sources might contain malware  or generic unwanted programs (Softonic is known to host plenty of installers and grayware that asks you to install other third-party programs). The authors should better clarify the origins of these data, or at least try to study the quality of the provided ground truth. Otherwise, the dataset might contain biases that reduce the fairness of the publication.

Author response:
>>Evaluating the quality of goodware data is a very good point, and we are actively trying to address this. We are in the process of getting access to the premium API of VirusTotal, and are planning to scan all the benign files to filter out any malicious file before the official release. Meanwhile, we are working on getting some preliminary statistics before the discussion ends.

Reviewer comment (Novelty):
**False statements.** The authors state that "it is difficult to add more than 10% of content". This statement is false, since adversarial malware attacks are automated through tools. Papers like [Demetrio et al. 2021a&b / Lucas et al. 2021] can increase the size more than 10% of the file size (Lucas et al. bound it to 5% just to not enlarge too much the input file). Lastly, Header Modification is not proposed by Nisi et al. 2021, but it is contained inside the SecML Malware library, inspired by the paper (that states which fields are not used by the loader anymore).

Author response:
>>In the last paragraph of subsection 6.2, we mentioned that – perturbing 200KB in 2MB file (=10%) is ‘challenging’ in a malware file. If that is the concern, we have rephrased it and toned it down to **‘considered as a sizeable modification’**. If you have further suggestions, let us know.   
>  
>*"Lastly, Header Modification is not proposed by Nisi et al. 2021, but it is contained inside the SecML Malware library, inspired by the paper (that states which fields are not used by the loader anymore)."*  
>>We are aware of the fact that – Nisi et. al. 2021 did not explicitly propose the header field modification attack but we wanted to mention the most relevant paper to the attacks so that interested readers can study them. We have changed *‘proposed by’* to *‘motivation from’* in our text for Nisi et. al. paper. If you have further suggestions regarding presentations, we are more than happy to incorporate them.

Reviewer comment (Novelty):
**Limitations and related work not addressed.** The paper does not discuss limitations of their methodology, by just saying that certification is a difficult problem to solve. Also, related work misses a preliminary (but unpublished) paper [1] that addressed the problem in the early months of 2023 (more than 6 months ago). It would be better to mention the fact that preliminary work on certification for malware detection are already there.

Author response:
>>We wanted to put a ‘Limitations’ section before the ‘Conclusion’. However, due to the page constraint, we could not do that. So, we have added a ‘Limitations’ section in our **Appendix (A.6)** and referred to it in our ‘Conclusion’ section.  
>  
>Also, related work misses a preliminary (but unpublished) paper [1] that addressed the problem in the early months of 2023 (more than 6 moths ago). It would be better to mention the fact that preliminary work on certification for malware detection are already there.  
[1] Certified Robustness of Learning-based Static Malware Detectors - https://arxiv.org/pdf/2302.01757.pdf"*  
>>Although ICLR policy tolerates such overlooks for preprints without peer reviews that are published after May 28 (https://iclr.cc/Conferences/2024/ReviewerGuide), we have discussed this recent paper in the ‘Related Work’ of our revision.

Reviewer comment (Reproducibility):
How you conducted the adversarial attacks? Which library did you use / how you obtained the code of attacks? Inside the provided material, it is not possible to test the attacks from Lucas et al.

Author response:
>>For most of the attacks we used the `secml-malware` Python library. Also, we built the DRSM framework on top of the secml-malware library so that it can be easily reproduced, extended, and evaluated against more attacks in future (mentioned in the footnote of page 2).   
>>  
>>We really appreciate your effort of checking our provided material and code. About Disp and IPR attack of Lucas et. al. – we collected the code implementation directly from the authors. Though we want to make everything of this work publicly available, we cannot do that for Disp and IPR attack. The white-box implementation of these attacks was kept private by the authors, and before access, we agreed to keep it private too.

Reviewer comment (Experiments):
Can the authors provide better information on the quality of the collected goodware?

Author response:
We have so far scanned randomly selected **1000 benign files** from our PACE dataset. For each file, we got results from 54~72 engines from the VirusTotal. So far, we have got only 1 file *(MD5 hash = cfa051242ce5d13f9fb588736c4601ec)* that was detected as malicious by 19 engines out of 67. The rest of the files were found as ‘benign’. So, even if we label a file ‘malware’ just because of getting detected by only 28% of the engines (which would be a conservative assumption), statistically, there will be 0.1% malicious files in our dataset. However, we understand that 1000 files do not represent the whole dataset, so, we are still in the process of scanning all files. And we will give you another update just before the discussion phase ends.

Reviewer comment (Writing):
There are a few grammatical issues throughout the paper - it needs a polish before publication, e.g.  
- "We will use it as base classifiers"  
- "potentially attributing to the issue"  
- "convolution neural network"  
- "there have been a large amount of work"  
- "to some extents"  
- "though it has been believed as a robust model"  
- "However, it's worth highlighting.." not sure However is right word.

Author response:
We really appreciate your effort in reading our paper thoroughly and pointing out the grammatical issues. We have corrected all of the mentioned ones. You can find them in blue text in the **updated version** of our submission. Let us know if you find anything else. We would love to have your feedback.

Reviewer comment (Theory):
X \subset [0,N-1] is wrong. [0,N-1] is a real-valued continuous interval. I think it means X is the set {0, 1, ..., N-1}.

Author response:
Thank you for pointing this out. We have addressed this in our revised version.

Reviewer comment (Presentation):
Would it be possible to indicate the values of \Delta on the x-axis of figure 3? Everything has been in terms of that up to this point.

Author response:
We have added another figure **(Figure 10 in the Appendix A.4)** that shows the certified accuracy in terms of $\Delta$. From this figure, it is interpretable that a smaller window size achieves higher certified accuracy. At the same time, we want to mention that – a smaller window allows attackers a smaller budget for perturbation. For example, if $\Delta =2$, in DRSM-4, the attacker can perturb up to 511K bytes, whereas in DRSM-24, it is 82K bytes. So, it might not be fair to compare these models just depending on the $\Delta$, and hence, we kept the Figure 3 showing accuracy with respect to perturbed bytes in the main text, and put the Figure 10 in the Appendix so that interested readers can see them and easily interpret the results. We would also love to have your feedback on this one.

Reviewer comment (Novelty):
The paper can be seen as applying an existing method (de-randomized smoothing) to a new domain (malware). Although the authors claim that it is “challenging” to adapt de-randomized smoothing for malware, I’m not convinced. The proposed method is a form of structured ablation, originally studied by Levine & Feizi (2020) for 2-d inputs with homogeneous base classifiers. The modification from 2-d to 1-d inputs and from homogeneous to heterogeneous base classifiers seems straightforward. Moreover, the proposed method has appeared in prior work in a more general form by Hammoudeh & Lowd (2023).

Author response:
>>We respectfully argue that the success of a ‘general’ method in a specific domain should not be taken for granted. An example will be the paper of Hammoudeh & Lowd (2023) mentioned in your review, which also resembles Levine & Feizi (2020): One contribution of Hammoudeh & Lowd (2023) is their study of partitioning strategies, in which they suggest using stridded input dimensions (pixels) is beneficial for vision tasks, potentially because information contained by adjacent pixels are redundant to some extent. In contrast, for the malware domain, due to its nature, some adjacent bytes are malicious only when they are combined, i.e., not considering adjacent bytes might end up representing a whole different instruction (or invalid instruction) that might not be malicious.

Reviewer comment (Novelty):
The paper claims to be “first to offer certified robustness in the realm of static detection of malware executables”. However there is prior work on this topic by Huang et al. (2023) which appeared on arXiv in January 2023. Their work considers a different threat model for malware: edit distance robustness rather than patch robustness.

Author response:
>>Although ICLR policy tolerates such overlooks for preprints without peer reviews that are published after May 28 (https://iclr.cc/Conferences/2024/ReviewerGuide), we have acknowledged and discussed this recent paper in the **‘Related Work’ of our revision**. Additionally, we want to point out that – along with the different threat model (that you already mentioned), this paper adapts the randomized smoothing scheme, which differs from ours.

Reviewer comment (Theory):
A characteristic feature of the malware domain is that inputs vary in length. However, it’s not clear to me how the proposed classifier architecture handles this. As an example, consider a 100 KB malicious file and a classifier with a maximum input length of 2 MB. For $n$ in the range 4–20, the malicious file fits within a single block, meaning it is passed to a single base classifier, while the remaining $n - 1$ base classifiers receive padding as input. Assuming the base classifiers predict “benign” for padding, the votes are $1$ “malicious” and $n - 1$ “benign” giving a prediction of “benign”. I wonder if I’m missing something here, because it seems the classifier is guaranteed to make false negative errors on small files, which are abundant according to Figure 6.

Author response:
>>This is a very good observation. To tackle this issue in our DRSM framework, we consider the votes (or predictions) from the base classifiers that get any input (except padding). We could consider all votes and solve this by adding an extra learnable layer (such as logistic regression) on all votes, but it would have hurt the non-differentiability property (eventually, certified robustness) of the whole framework.

Reviewer comment (Writing):
I found the description of the threat model and certificate unclear. Section 3.2 states that the attacker is allowed to “modify or add any bytes in a contiguous portion”. I understand that “modify” means overwrite or replace, but it’s not clear what “add” means in this context. For instance, “add” could mean “insert” or “append”, or it may mean “increment or decrement by some amount”. The mathematical description $x’ = x + \delta$ implies the original sequence $x$ is additively perturbed by $\delta$ (which is undefined), but this seems to be at odds with the earlier description. Reading between the lines, my understanding is that the certificate covers a contiguous chunk of the original file being overwritten, which may include some bytes being appended to the end of the file. This should be precisely stated somewhere. The current definition of the certificate in Section 5 is in terms of ablated sequences – it would be helpful to translate this to the input space.

Author response:
>>Thanks for your constructive feedback and asking for the clarification of the term ‘add’. We have rephrased a few sentences in the second paragraph of our threat model **(subsection 3.2)**. With the term ‘add’, we meant the attacker can ‘insert’ or ‘append’ any bytes; not ‘increment / decrement’. Additionally, we have mentioned that it has to be bounded in a contiguous portion. Also, we apologize if the mathematical description looked confusing. We have reformed that into a simple sentence now for better clarity. 
Additionally, we have rephrased a few sentences in the **third paragraph of the threat model**, and mentioned what type of attack falls within our threat model. Let us know if you have further suggestions about the presentation. We would appreciate your feedback.

Reviewer comment (Experiments):
It’s great that the authors are planning to release the PACE dataset. However, the current description of the collection process is a bit light on detail. It would be helpful to describe how binaries were selected from the various sources. For instance, was there a preference for recent binaries? Are the binaries for a single platform (e.g., Windows x64) or multiple platforms? How do you know the binaries are benign?

Author response:
>>Thanks for appreciating our plan to release the PACE dataset. The sources of this dataset are given in **Table 2**, and we crawled these websites to download the benign files. We downloaded them in August, 2022 and there was no preference for recent binaries. Yes, the binaries are for a single platform (windows). Evaluating the quality of goodware data is a very good point, and we are actively trying to address this. We are in the process of getting access to the premium API of VirusTotal, and are planning to scan all the benign files to filter out any malicious file before the official release. Meanwhile, we are working on getting some preliminary statistics before the discussion ends.

Reviewer comment (Theory):
Section 3 states that the input vector fed into the network “has to be of a fixed dimension”. This is not true in general, and I don’t believe it’s true for MalConv. I believe it is possible to support arbitrary length inputs in modern frameworks such as PyTorch by specifying `None` for the size of the dimension.

Author response:
>>With that sentence, we wanted to mention the original implementation of MalConv that was done for 2MB of file size.

Reviewer comment (Theory):
Section 3.1 states that models like EMBER and GBDT “can work only on feature vectors”. I’m a bit puzzled by this statement. When composed with their feature extractors, these models must be able to operate on raw binaries, otherwise they would be useless as malware detectors?

Author response:
>>We are aware that EMBER, GBDT can work as malware detectors that require an extra feature extraction step unlike our work, and we apologize for the misunderstanding. We have rephrased that sentence in **subsection 3.1**.

Reviewer comment (Experiments):
Section 5 states that vision-oriented ablation techniques such as masking and block ablations are infeasible for byte sequences. However I can’t see why these wouldn’t work on 1d sequences? It seems feasible to mask bytes or ablate 1d blocks.

Author response:
>>An instruction is converted to multiple bytes that are usually contiguous, and masking a random byte might end up changing the meaning of an instruction or in the worst case, an invalid instruction. Block ablation was proposed for 2D input, it is not clear how it can be applied for 1D input. (We also update our manuscript with a recap of de-randomized smoothing in vision tasks in Appendix A.2.).

Reviewer comment (Writing):
Section 7 states MalConv NonNeg “has been believed as a robust model for a long time”. It would be good to include a citation for this claim.

Author response:
>>We see how this statement might be controversial. We have removed it from **section 7**.

Reviewer comment (Experiments):
The proposed "window ablation" strategy seems to be applicable only to scenarios where perturbations are clustered together (similar to patch attacks in the CV domain). However, many adversarial attack methods for malware disperse the inserted perturbations across various locations within the software. It remains unclear whether the proposed method would still be effective in such cases. Update: this has now been addressed.

Author response:
We really appreciate your concise review. 

Yes, you are right. The proposed method is ‘theoretically’ for the attacks that can modify or add in a contiguous region. And we also agree that there are many adversarial attacks that do not follow this and can modify/add in multiple regions. So, in this work, we actually evaluated our proposed model DRSM against such attacks, because we believe that – only ‘theoretical robustness’ is not enough for a security-critical application like malware detection. In **section 7 ‘Empirical Robustness Evaluation’**, we considered in total of 9 attacks where 5 of them can modify multiple regions in a file. In **Table 4**, we listed all these attacks with their short description and alignment with our threat model. For example, **Slack append, Header field modification, and Gamma attack** can perturb multiple regions in a malware file, and recently proposed **Disp and IPR** attacks can modify a file on the instruction level and are not limited to a certain region. 

We used these attacks to compare our DRSM with the baseline model ‘MalConv’ and its more robust variant ‘MalConv (NonNeg)’. The results are shown in **Figures 4 and 5**.

Reviewer comment (Experiments):
The construction of many malware samples follows a piggyback approach, where the majority of the software consists of benign code, with only a small portion exhibiting malicious behavior. That’s to say most of the ablated sequences will be given a benign label. The algorithm proposed by the author may result in false negatives for such malware samples. Update: this has now been addressed.

Author response:
This is a very good observation, and it is true that most of the contents in a malware file are actually benign. This argument actually aligns with our result too. We found that – if we increase the number of windows (decreasing the length of ablated sequences), the probability of an ablated sequence getting classified as ‘benign’ gets higher because a smaller ablated sequence would cover less content, and hence, less/no malicious content. As a result, for the higher number of windows ($n$), DRSM models have less standard accuracy, and we showed this in Table 3 and discussed this in subsection 6.1. For example, **DRSM-4 has 98.18%** standard accuracy, whereas **DRSM-24 achieves 90.24%**. The goal of this paper was to – find a balance between standard accuracy and robustness.

Reviewer comment (Experiments):
It is essential to provide a comparative analysis of the author's proposed defense method against existing adversarial example defense techniques for malware. Update: this has now been addressed.

Author response:
The existing defenses for adversarial malware can be broadly divided into two – Non-negative classifier, and Adversarial training. We have already included the former one, **‘MalConv (NonNegative)’**, in our work and compared it in terms of standard accuracy, certified accuracy, and empirical robustness in **Table 3 and Figures 4, 5 ( and Appendix A.3)**. We also want to emphasize that – recent work by Lucas et. al. [1] has already shown that – in most cases, the adversarially trained models (the second defense) do not provide good robustness against other attacks. Moreover, such defense compromises the standard accuracy too, for example, Lucas et. al. showed training the model adversarially on Kreuk-0.01 degraded the true positive rates to 84.4% ~ 90.1%.  We have discussed this in the second paragraph of **section 2 (Related Work)**. 

[1] Adversarial training for {Raw-Binary} malware classifiers. https://www.usenix.org/conference/usenixsecurity23/presentation/lucas

Reviewer comment (Theory):
Insufficient theoretical analysis of certified robustness when facing multiple malicious ablated sequences. Specifically, the authors assume that an attacker generates a byte perturbation of size $p$ and can modify a maximum of $\lceil \frac{p}{w} \rceil + 1$ ablated sequences. However, if the attacker simultaneously inserts multiple adversarial code segments at different locations, how will this impact the certified robustness? The authors should offer a more in-depth theoretical analysis of this scenario. Furthermore, the authors should establish the relationship between the window size ($w$) and the resulting certified robustness. For example, does a smaller window size lead to improved certified robustness?

Author response:
>Thanks for such a detailed response. We appreciate your effort in reading our paper thoroughly. 
>
>As the de-randomized smoothing approach was borrowed from the computer vision domain, we followed the same threat model where the attacker can add a patch of a specific size (add or modify byte sequence of $p$ size, in our case). In our theoretical study, we wanted to keep it as similar as the CV domain. At the same time, we also understand that the malware domain does not work like that, and your point on – *"the attacker simultaneously inserts multiple adversarial code segments at different locations"* is totally valid and we agree with that. Therefore, we empirically evaluated the robustness of our DRSM approach against such attacks that can insert multiple adversarial code segments **(section 7)**. To be specific, we have included attacks, namely **Slack Apppend, Header Field Modification, Gamma,** where the attacker can include or modify at multiple places, and stronger attacks, namely **Disp, IPR,** where the attacker can even modify on the instruction level.  **Table 4** includes the list of attacks we experimented with in this paper where the column ‘Threat Model’ indicates their alignment with our threat model. In Figures 4 and 5, we showed the attack success rates for these attacks. Also, we have included how all these attacks work and were implemented in **Appendix A.4**.

Reviewer comment (Presentation):
In my view, there appears to be a contradiction between Figures 1 and 2. As discussed in Section 5, malicious ablated sequences are expected to influence their respective base classifiers, and the predicted winning class should be "benign." However, in Figure 1, the predicted winning class is labeled as "malware,". It confuses me. If this work indeed presents a more robust framework that prevents attackers from generating adversarial examples, then the winning class should ideally be "benign." However, if the winning class is consistently benign, it suggests that the proposed framework might miss detecting certain malware instances, creating a contradiction.

Author response:
>*“If this work indeed presents a more robust framework that prevents attackers from generating adversarial examples, then the winning class should ideally be "benign."”* – we want to mention that – this work does not prevent attackers from generating adversarial examples. We proposed a framework (DRSM) on top of an already existing classifier that can improve its robustness for adversarial malware; our goal was not to stop malware authors from generating them. 
>
>In Figure 1, we tried to show the fundamental difference between the original base classifier (MalConv) and our method (DRSM) with a toy example in a simple way. Here, for DRSM, the adversarial malware gets ablated into 3 non-overlapping windows (or sequences) and generates 3 different predictions. Since the perturbation impacted only one window (the middle portion in the file written with red font), DRSM gave the wrong prediction (benign) only for that one. But for the rest of the windows (the first and third portion in the file written with black font), DRSM correctly classifies them as ‘malware’. As a result, ‘malware’ (2) wins against ‘benign’ (1) in voting, and DRSM gives the final output as ‘malware’.

Reviewer comment (Experiments):
The absence of comparisons with a broader range of real-world antivirus engines accessible via VirusTotal is notable. It would be beneficial, for instance, to determine how many antivirus engines effectively identify the malware and test cases as malicious.

Author response:
>The main focus of this paper was to propose a better defense model for adversarial attacks, not the attacks themselves. Moreover, we evaluated our models against 9 different attacks and there is a page limitation. So, we just kept the gist of the attacks in our main paper. **Table 4** lists all 9 attacks with their short description, and alignment with our threat model and settings. Additionally, in **Appendix A.4**, we included details for all of these attacks and their implementation in this work. Also, all of these attacks have already been proposed in published prior works showing that they do not compromise executable files, and we cited them so that any specific details about them can be retrieved if necessary. And some of these attacks (such as Disp, IPR, GAMMA, etc.) had already been tested against VirsuTotal and were found successful. We also want to mention that -- Disp, IPR attack (Lucas et. al. (2021)) reported that they can evade VirusTotal in 49%-53%, whereas in DRSM, these attacks could evade 42% and 9.50% cases, respectively, even for our weakest model (DRSM-4).

Reviewer comment (Reproducibility):
Another deficiency is the absence of specific details regarding the process of compromising executable files. The authors should provide a more comprehensive explanation of how to generate adversarial examples within the problem-space[Ref-1], which encompasses defining a comprehensive set of constraints on available transformations, preserving semantics, ensuring robustness to preprocessing, and maintaining plausibility.

Author response:
>The main focus of this paper was to propose a better defense model for adversarial attacks, not the attacks themselves. Moreover, we evaluated our models against 9 different attacks and there is a page limitation. So, we just kept the gist of the attacks in our main paper. **Table 4** lists all 9 attacks with their short description, and alignment with our threat model and settings. Additionally, in **Appendix A.4**, we included details for all of these attacks and their implementation in this work. Also, all of these attacks have already been proposed in published prior works showing that they do not compromise executable files, and we cited them so that any specific details about them can be retrieved if necessary. And some of these attacks (such as Disp, IPR, GAMMA, etc.) had already been tested against VirsuTotal and were found successful. We also want to mention that -- Disp, IPR attack (Lucas et. al. (2021)) reported that they can evade VirusTotal in 49%-53%, whereas in DRSM, these attacks could evade 42% and 9.50% cases, respectively, even for our weakest model (DRSM-4).

Reviewer comment (Experiments):
The rationale behind the design choice is unclear. Why have the authors chosen MalConv as the baseline classifier? There are numerous alternative models that can serve as the baseline classifier, such as LGBM, RF, and SVM. The authors should consider evaluating their framework with these alternative baseline classifiers.

Author response:
>We want to emphasize that CNN-based models like MalConv can take the whole raw bytes as input whereas models like LGBM, RF, or SVM require feature engineering (e.g., byte entropy), and they cannot take raw bytes as input. These features are known to be specifically targeted by adversaries to avoid detection, which makes "'being able to take the whole raw binary" attractive. However, this new capability introduces a risk of adversarial attacks too, which motivates us to position our defense for CNN-based models. As MalConv is one of the state-of-the-art CNN based static classifiers, it was a suitable choice to evaluate our framework. However, note that our window-ablation scheme is agnostic of the base classifier as it operates at the input level. As a result, it could be applied to any detector that consumes raw bytes.

Reviewer comment (Presentation):
If the winning class of Fig. 2 is "benign"?

Author response:
>Figure 2 is a generalized Figure where the input file can be anything (malware or benign). The green check stands for a correct prediction, whereas the red cross stands for a wrong prediction. At the end, the bars stand for the number of correct vs wrong predictions.
For example, let us assume that the input file is an adversarial malware, and the perturbed region of that malware falls into one of the ablated sequences of DRSM (shown with a red small block in ablated sequences). So, the base classifier misclassifies that sequence (shown with a red cross) but classifies the rest of the sequences correctly (shown with green checks). Hence, the winning class is still ‘malware’ (shown with the green bar in ‘count class prediction’). We apologize if the figure is confusing. We have added a small description in the caption. If there is any scope to make the presentation better, let us know. We would love your feedback.

Reviewer comment (Experiments):
Does a smaller window size lead to improved certified robustness?

Author response:
>Thanks for asking about the relationship between window size and certified accuracy. Yes, a smaller window size leads to a better certified accuracy. And we discussed this in the **last paragraph of subsection 6.2**. We mentioned that – *“By analyzing Table 3, we can see that $n$ has a positive and negative correlation with certified and standard accuracy, respectively. While DRSM-24 provides the highest certified accuracy (53.97%), it has the lowest standard accuracy (90.24%).”*  The Table 3 shows the certified accuracy for the same $\Delta (=2)$ for each model variant, and notably, DRSM-24 (with the smallest window size) achieves the highest certified accuracy (53.97%) whereas DRSM-4 (with the highest window size) achieves the lowest certified accuracy (12.2%). Additionally, we have added another figure **(Figure 10 in the Appendix A.4)** that shows the certified accuracy in terms of $\Delta$. From this figure, it is interpretable that smaller window size achieves higher certified accuracy. At the same time, we want to mention that – a smaller window allows attackers a smaller budget for perturbation. For example, if $\Delta =2$, in DRSM-4, the attacker can perturb up to 511K bytes, whereas in DRSM-24, it is 82K bytes. So, it might not be fair to compare these models just depending on the $\Delta$, and hence, we kept the Figure 3 showing accuracy with respect to perturbed bytes in the main text, and put the Figure 10 in the Appendix. We would appreciate your feedback too.