From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The paper performs an empirical investigation on why existing importance-aware adversarial training (AT) methods fail and proposes a fix for this. The main empirical novelty of the paper would be that the method is the first importance-aware AT that improves on the AutoAttack benchmark compared to the standard AT. After the active discussion between the author and reviewers, two reviewers ended on the positive side, and the other two were negative about the acceptance of this paper. Despite the positive reviews, AC agrees with multiple reviewers that the technical significance of the paper is somewhat weak due to (a) the lack of convincing justification and (b) its incremental improvements over the base AT algorithms. Furthermore, given the overall marginal gains, AC thinks (c) the lack of discussion (or clarification) on the adaptive attack can be another weakness. For example, the authors should design a new type of attack for the SOVR objective (such as a logit margin-based attack) to fully convince the robustness of SOVR [1]. 

[1] Tramer et al., “On Adaptive Attacks to Adversarial Example Defenses”, NeurIPS 2020
justification_for_why_not_higher_score: Although the paper performs valuable explorations on the topic, AC thinks the reasons to reject slightly outweigh the reasons to accept at the current form. 
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Thank you for follow-up comments
comment: Thank you very much for your additional response that explains your concerns! We are very glad to understand your concern in detail.

> Regarding the strong assumption in theory development, I wanted to mention to the assumption about Lipschitz constant of the $k$-th logit function.

With the greatest respect, we would like to point out that our main theoretical results (Lemmas 4.3 and 4.4 and Theorems 4.1 and 4.5) do not use the assumption about Lipschitz constants. We would like to know which theorem or lemma you think this assumption is an issue for.
 
>  The one-versus-the-rest loss is very popular and widely used in many other works [1,2] and so on.

We appreciate that you inform us of previous studies using OVR and will cite them in our paper. 

We present a novel advantage of OVR for neural networks compared with cross-entropy: [1] and [2] do not discuss the adversarial robustness and its effect on logit margins. Our paper uses OVR for another motivation and provides new theoretical analyses. We believe that it is a novel contribution to revisit an old method through the discovery of new values and new effects with the theory and experiment.

We mostly agree that OVR is an old well-used method but respectfully could not agree that OVR is very popular in deep learning because the recent paper [2] cited only one related study (Padhy et al.)[b] that uses OVR to train neural networks. [1] uses OVR to train SVM but does not use OVR to train neural networks. [1] uses a meta-learned neural network as a feature-extractor for the SVM, and the loss function of the meta-learned neural network is cross entropy. [2] uses OVR for open-set domain adaptations. While we use OVR to increase logit margins for robustness, [2] leverages OVR to consider unknown classes in the target domain. Thus, [2] does not discuss robustness and logit margins and does not provide theoretical results.

From: ICLR.cc/2023/Conference/Paper2328/Reviewer_G3JG
title: Thanks for your response
comment: Thanks for responding my questions. 

Regarding the strong assumption in theory development, I wanted to mention to the assumption about Lipschitz constant of the $k$-th logit function. 

My main concern is about the novelty of this work. The one-versus-the-rest loss is very popular and widely used in many other works [1,2] and so on.

[1] Lee, K., Maji, S., Ravichandran, A. and Soatto, S., 2019. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10657-10665).

[2] Saito, K. and Saenko, K., 2021. Ovanet: One-vs-all network for universal domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9000-9009).

The paper only adopt the one-versus-the-rest loss for adversarial machine learning, which seems not sufficiently significant for a venue like ICLR. Although acknowledging the authors in writing a good paper, I am keen on my current score.

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Thank you for your response!
comment: We are glad to see that your concerns about clarity seem to be addressed! Thank you again for your work on our paper.

With the greatest respect, we would like to emphasize one point: our contribution is not only the performance of SOVR, but also the fact that we have clarified the issues of existing methods based on the distributions of logit margins, and have theoretically revealed the difference in the behavior of logit margins between OVR and CE (These contributions are pointed out by Reviewer HRBR and Fx8Y). We would be very pleased if these contributions could make a slightly positive impression on our paper.

From: ICLR.cc/2023/Conference/Paper2328/Reviewer_aV6p
title: Thanks for the responses
comment: I appreciate your detailed response, while I think the improvement is marginal. So I decide not to update my score.

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Thank you for your response!
comment: We greatly appreciate additional comments after reading our feedback! Thank you again for sparing the time and effort to review our paper.
> I still believe that the idea is incremental and the theory needs to make strong assumptions.   
> The assumption in Eq. (4) is extremely infeasible and simple compared to real-world uses of the CE and logit margin losses. 

If possible, we would like to understand your concern about the assumptions better. Could you tell us what "the assumption in eq. (4)" is in your comment? As in our first comment, we do not add any assumptions in equation (4) on page 2 because equation (4) is the exact loss function used in the previous methods.  We would be very glad to hear your concern about the assumptions.

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Gentle Reminder
comment: Dear Reviewers and ACs

We greatly appreciate the reviewers' efforts in reviewing our paper. We thank the ACs very much for encouraging the discussion about our paper and our feedback.

Following the reviewers' comments, we revised our paper and answered questions in Discussion Stage 1. Since the discussion period is until Dec. 12, we would appreciate it if you could inform us of the remaining concerns. We would be delighted to have further discussions about our paper before the deadline.

Best regards,

From: ICLR.cc/2023/Conference/Paper2328/Area_Chair_5BCD
title: Following up on authors’ response and discussion
comment: Dear Reviewers,

Thank you very much again for performing this extremely valuable service to the ICLR community.

Please check the authors’ response and leave comments if you have not done it.

Best,

AC

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Evaluation of the combination between SOVR and [c]
comment: We are pleased to report the results of the full experiment.

We finished the evaluation of the combination between SOVR and [c]
For this evaluation, we used the [code](https://github.com/imrahulr/adversarial_robustness_pytorch) provided by the original authors and only changed the loss function for our method.
Following [c], we used label smoothing for cross-entropy in TRADES, SOVR, and TSOVR. 
Whereas [c] reported the best result, we trained models three times and calculated averages and standard deviations for reproducibility.
We obtained the following results:

**Robust Acc. against Auto-Attack (WRN-28-10) when using 1 M synthetic data**
|WRN28-10|AT|TRADES|SOVR|TSOVR|
|-|-|-|-|-|
|Robust Acc.|59.3$\pm$0.9|60.13$\pm$0.5|60.35$\pm$0.5|**60.65$\pm$0.07**|
|Clean Acc.|**88.46$\pm$0.05**|86.1$\pm$0.2|88.42$\pm$0.07|86.81$\pm$0.01|
|Robust+Clean|147.76$\pm$0.9|146.23$\pm$0.5|**148.77$\pm$0.5**|147.46$\pm$0.07|

In this table, TSOVR achieves the highest robust acc., and SOVR achieves the highest summation of robust and clean accuracies.

We thank you for your suggestion, and we appreciate that the new results strengthen our claim.

[c] Rebuffi et al., "Fixing Data Augmentation to Improve Adversarial Robustness" arXiv 2021

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Thank you for the response
comment: We thank you for your reply and are glad to hear your concern! We agree with you that the further evaluation, possibly using additional data, will strengthen our paper and are considering an evaluation of the SOVR using generated data. 

Our quick experimental result based on [c] is as follows:

Robust Acc. against Auto-Attack (WRN-28-10) when using 1 M generated data 
||TRADES|SOVR|
|-|-|-|
|Clean Acc.|85.94|88.47|
|Robust Acc.|60.22|60.56|

We can see SOVR achieves a good trade-off with high robust acc, which is consistent with the other results. The generated dataset is obtained from [this repository](https://github.com/deepmind/deepmind-research/tree/master/adversarial_robustness). 

Due to the computation cost, we could not finish the full experiments until the deadline of Phase 1. While we are unable to submit the revision during the Phase 2 discussion, we will report the results via the comment form when we will obtain them.

[c] Rebuffi et al., "Fixing Data Augmentation to Improve Adversarial Robustness" arXiv 2021

From: ICLR.cc/2023/Conference/Paper2328/Reviewer_Fx8Y
title: Reply after rebuttal
comment: I thank the authors for the response and additional experiments.

I agree that the comparison among methods should be carried out with the same training data. However, it'd be important to test the performance of SOVR in combination with the latest improvements introduced for adversarial training (besides changes to the loss function), possibly including using additional data for training.

In the reported comparison, I think one could improve the robustness achieved by HAT (maybe matching that of TSOVR) by tuning its hyper-parameters: in fact, it has clean accuracy significantly higher than TSOVR, which could be reduced to improve robustness.

Overall, I'd keep the original score.

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Response to Reviewer Fx8Y
comment: We appreciate your supportive comments and insightful comments based on your careful reading. Following your comments, we update the figures of histograms in the revision, and we are glad that they improve the quality of our paper. 

> While there's still a difference in the distribution of the margins between naive adversarial training and the importance-aware methods, I think the plots for the models with early stopping should be reported in the main part to give a more complete picture of the phenomenon the paper analyzes.

Following your comments, we added the histograms by early stopping in Figs. 2 and 4 of the revision. They make our paper more complete: by comparing two figures, we can guess how logit margins grow in training besides the difference in the distribution of logit margin losses among defense methods. 

We first showed the results at the last epoch in the main of the previous submission because importance-aware methods and SOVR only change the loss functions from AT. We aim to evaluate the pure effect of the loss functions by using their optimal points. Even so, the early stopping is usually used in adversarial training as you say, and we really like updated figures!

> The only direct comparison to TRADES is with WRN on CIFAR-10, and the two methods achieve similar results.

We additionally compared TRADES with SOVR and TSOVR on other settings  (the same tables in General response):

**Table A Robust Acc. against Auto-Attack**
| |TRADES|TSOVR|SOVR|
|-|-|-|-|
|C10 (RN) |48.8 $\pm$ 0.3|**49.8 $\pm$ 0.1**|49.4 $\pm$ 0.3|
|C10 (WRN) |52.9 $\pm$ 0.3|**53.3 $\pm$ 0.1**|53.1 $\pm$ 0.2|
|C10 (WRN)+AWP |55.6 $\pm$ 0.4|**56.4 $\pm$ 0.2**|56.03 $\pm$ 0.07|
|SVHN |49.4 $\pm$ 0.3|**49.87 $\pm$ 0.02**|48.5 $\pm$ 0.4|
|C100 |23.2 $\pm$ 0.1|**24.61 $\pm$ 0.07**|24.3 $\pm$ 0.2|

**Table B Clean Acc.**
| |TRADES|TSOVR|SOVR|
|-|-|-|-|
|C10 (RN) |**82.5 $\pm$ 1**|81.4 $\pm$ 0.2|81.9 $\pm$ 0.2|
|C10 (WRN) |84.2 $\pm$ 0.7|83.1 $\pm$ 0.2|**85.0 $\pm$ 0.2**|
|C10 (WRN)+AWP |84.9 $\pm$ 0.5|84.9 $\pm$ 0.4|**86.9 $\pm$ 0.24**|
|SVHN |88.7 $\pm$ 1|88.5 $\pm$ 0.5|**90.0 $\pm$ 1**|
|C100 |**53.8 $\pm$ 0.3**|51.6 $\pm$ 0.2|52.1 $\pm$ 0.8|

**Table C Trade-off: Robust Acc. + Clean Acc.**
| |TRADES|TSOVR|SOVR|
|-|-|-|-|
|C10 (RN) | **131.3 $\pm$ 1** | 131.2 $\pm$ 0.3 | **131.3 $\pm$ 0.3**|
|C10 (WRN) |137.1 $\pm$ 0.7|136.4 $\pm$ 0.2 |**138.1 $\pm$ 0.2**|
|C10 (WRN)+AWP |140.5 $\pm$ 0.6|141.3 $\pm$ 0.4|  **142.9 $\pm$ 0.2**|
|SVHN |138.1 $\pm$ 1|138.4 $\pm$ 0.5|**138.5 $\pm$ 1**|
|C100| **77.0 $\pm$ 0.3** | 76.2 $\pm$ 0.2|76.4 $\pm$ 0.8|

In the tables, TSOVR achieves the best robust accuracy. On the other hand, SOVR achieves the best trade-off in most cases if the trade-off is defined as the summation of robust and clean accuracies.

We added the above results in Appendix F.9 and the following statement in Section 6.2: "We compare them on other datasets in Appendix F.9." ("them" corresponds to AT, SOVR, TSOVR, and TRADES).

> While SOVR outperforms importance-aware methods, a comparison to other methods (see e.g. https://robustbench.github.io/) for obtaining robust models is missing. 

We would respectfully like to inform you that top rankers in RobustBench use huge additional data or are not loss-function-based approaches, unlike SOVR. Specifically, top rankers use 1M synthetic images for CIFAR10, which only has 0.05 M images. If the defense method does not change the loss function, it is used with SOVR, like AWP (Table 2), which is one of the best defense methods with WideResNet34-10 without additional data and augmentations in RobustBench.

Even so, we are also interested in the relative performance compared with recent loss-function-based methods. We evaluated HAT [b] because it changes the loss function, like SOVR. While the motivation of HAT is to achieve a good trade-off between robustness and clean accuracy, HAT improves robustness more than TRADES in [a], and it is located at a high rank in RobustBench. 

**Table G Comparison with HAT (Robust Acc. against AutoAtatck)**
| |HAT|SOVR|TSOVR|
|-|-|-|-|
|C10 (RN) |48.43 $\pm$ 0.04|49.4 $\pm$ 0.3|**49.8 $\pm$ 0.1**|
|C10 (WRN) |52.6 $\pm$ 0.1|53.1 $\pm$ 0.2|**53.3 $\pm$ 0.1**|
|SVHN |32.3 $\pm$ 10|48.5 $\pm$ 0.4|**49.87 $\pm$ 0.02**|
|C100 |22.4 $\pm$ 0.2|24.3 $\pm$ 0.2|**24.61 $\pm$ 0.07**|

**Table H Comparison with HAT (Clean Acc.)**
| |HAT|SOVR|TSOVR|
|-|-|-|-|
|C10 (RN) |**84.3 $\pm$ 0.3**|81.9 $\pm$ 0.2|81.4 $\pm$ 0.2|
|C10 (WRN) |**86.4 $\pm$ 0.4**|85.0 $\pm$ 0.2|83.1 $\pm$ 0.2|
|SVHN |**91.5 $\pm$ 0.1**|90.0 $\pm$ 1|88.5 $\pm$ 0.5|
|C100 |**57.4 $\pm$ 0.4**|52.1 $\pm$ 0.8|51.6 $\pm$ 0.2|

We used the same training setting as AT, and hyper-parameters of HAT are $\beta=2.5$ and $\gamma=0.5$ following [b].
HAT achieved better trade-off, which is consistent with its motivation. However, the robust accuracy of HAT is lower than SOVR. 

[b] Rahul R., et al. "Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off" ICLR2022

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Response to Reviewer aV6p Part 1/2
comment: We would like to thank you for sparing your time and your useful comments. Following your comments, we add the comparison to TRADES on other datasets (Appendix F.9 in the revision). We would be happy if the following explanations help to address your confusion. If our explanations are missing the points to clarify our motivations and claims, we would be very glad to hear your concerns a little more.
## Confusion about claims about logit margins  
> I'm confused about "On the other hand, logit margins of importance-aware methods concentrate near zero, and thus, importance-aware methods reduce the logit margins of easy samples. This implies that the weighted cross-entropy used in importance-aware methods is not very effective in increasing logit margins. To increase the logit margins of difficult samples, ...". 

This confusion might be caused by signs (+, -) of logit margin losses. Before responding to this question, we would like to clarify the notation and terminology in our paper. We define the logit margin loss as $\ell_{LM}=z_{k*}-z_y$ and define the logit margin as $|\ell_{LM}|$, i.e., logit margin losses have negative values when the model correctly classify samples. We use "increase" and "reduce" only for the absolute values $|\ell_{LM}|$, to avoid confusion. And thus, "increasing logit margins" means "increasing absolute values of logit margin losses with negative signs, i.e., increasing $|\ell_{LM}|$ for $\ell_{LM}<0$". In the following explanations, we are considering that data points satisfy $\ell_{LM}<0$ for simplicity.

> First, why the observation of "importance-aware methods reduce the logit margins of easy samples" can derive "importance-aware methods is not effective in increasing logit margins"? 

After minimizing the loss of each importance-aware method, its histogram of logit margin losses has only one peak around zero ($|\ell_{LM}|=0$, Figs.2 (b)-(f)). On the other hand, AT has two peaks in its histogram (Fig.2(a)).
The left peak is located around -10 ($|\ell_{LM}|=10$ with $\ell_{LM}<0$). We call data points on the left peak "easy samples", which correspond to the data points far away from the decision boundary. From this observation, while AT increases $|\ell_{LM}|$ for easy samples (satisfying $\ell_{LM}<0$), importance-aware methods do not increase $|\ell_{LM}|$ of easy samples. Both AT and importance-aware methods do not increase logit margins of difficult samples well. These observations indicate that importance-aware methods are less effective in increasing logit margins $|\ell_{LM}|$ than AT. 

> Second, why do you want to increase the logit margins of difficult samples? Do the large logit margins indicate the examples are well learned?

This is because large logit margins indicate that models are "robust near the samples" rather than "well learning them" as in Section 3.1. From Proposition 3.2 and definition 3.1, $\ell_{LM}=z_{k*}-z_y$ is desired to be smaller than $-(L_{\bar{k}}+L_y)\varepsilon$ to classify samples correctly under any perturbation $\mathbf{\delta}$ such that $|| \delta|| \infty$ $\leq$ $\varepsilon$. Since $\ell_{LM}=z_{k*}-z_y$ has negative values on correctly classified samples and $(L_{\bar{k}}+L_y)\varepsilon$ is positive, $\ell_{LM}=z_{k*}-z_y < -(L_{\bar{k}}+L_y)\varepsilon$ means $|\ell_{LM}| > (L_{\bar{k}}+L_y)\varepsilon$. Thus, logit margins $|\ell_{LM}|$ should be larger for better robustness. From this and the histograms ($|\ell_{LM}|$ concentrates around zero in Figs.2 (b)-(f)), we can figure out that importance-aware methods seem to fail to improve the robustness around both difficult and easy samples. Thus, we aim to increase logit margins of difficult samples with preserving logit margins of easy samples to improve the robustness around difficult samples.

> And the proposed method is to minimize the logit margins of "easy samples" if I didn't misunderstand.

SOVR does not minimize logit margins of easy samples (Fig. 4). SOVR increases logit margins of "difficult samples" with preserving the effect of AT on easy samples by switching losses. As shown in Eq. (18), we apply cross-entropy to samples in $\mathbb{S}$, which correspond to easy samples ($\mathbb{S}$ is a set where samples have small $\ell_{LM}$, i.e., the set of samples that has large $|\ell_{LM}|$ with $\ell_{LM}<0$). On the other hand, for difficult samples $\mathbb{L}$, we increase logit margins by using OVR, which theoretically increases logit margins (Theorem 4.5). Since increasing logit margins of easy samples causes overfitting (Fig.5), SOVR attempts to shift only difficult samples from the right peak to the left peak in AT.

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Response to Reviewer aV6p Part 2/2
comment: ## Comparison with TRADES 
> What about TRADES perform in Figure 2? Can the histogram of TRADES also be divided into two peaks?

Partially Yes. TRADES with WRN has two peaks but they are very close to each other (Fig. 12 (b) in Appendix E.1 of the revision). This might be because the objectives for parameters and for adversarial examples are different in the TRADES loss. This difference of logit margins might cause the good performance of TRADES. Thank you for  raising the interesting point.

> In Table 1, what about the results of TRADES?

We evaluated TRADES and TSOVR for all settings as shown in the following tables (the same as tables in General response).
They show that the order of robust accuracy is TSOVR>SOVR>TRADES except for SVHN, and TSOVR achieves the best robust accuracy on all datasets.

**Table A Robust Acc. against Auto-Attack**
| |TRADES|TSOVR|SOVR|
|-|-|-|-|
|C10 (RN) |48.8 $\pm$ 0.3|**49.8 $\pm$ 0.1**|49.4 $\pm$ 0.3|
|C10 (WRN) |52.9 $\pm$ 0.3|**53.3 $\pm$ 0.1**|53.1 $\pm$ 0.2|
|C10 (WRN)+AWP |55.6 $\pm$ 0.4|**56.4 $\pm$ 0.2**|56.03 $\pm$ 0.07|
|SVHN |49.4 $\pm$ 0.3|**49.87 $\pm$ 0.02**|48.5 $\pm$ 0.4|
|C100 |23.2 $\pm$ 0.1|**24.61 $\pm$ 0.07**|24.3 $\pm$ 0.2|

**Table B Clean Acc.**
| |TRADES|TSOVR|SOVR|
|-|-|-|-|
|C10 (RN) |**82.5 $\pm$ 1**|81.4 $\pm$ 0.2|81.9 $\pm$ 0.2|
|C10 (WRN) |84.2 $\pm$ 0.7|83.1 $\pm$ 0.2|**85.0 $\pm$ 0.2**|
|C10 (WRN)+AWP |84.9 $\pm$ 0.5|84.9 $\pm$ 0.4|**86.9 $\pm$ 0.24**|
|SVHN |88.7 $\pm$ 1|88.5 $\pm$ 0.5|**90.0 $\pm$ 1**|
|C100 |**53.8 $\pm$ 0.3**|51.6 $\pm$ 0.2|52.1 $\pm$ 0.8|

Since (Wang & Wang, 2022) reported that TRADES degrades the performance of SEAT, we did not evaluate SEAT + TRADES.
In the revision, we added the above results in Appendix F.9 and wrote the following statement in Section 6.2: "We compare them on other datasets in Appendix F.9." ("them" corresponds to AT, SOVR, TSOVR, and TRADES in this statement).

We are happy to add these new tables that make our paper strong.

> In Table 2, I'm afraid that the results can't convince me that the proposed method can increase the robustness. Compare with AT+AWP and TRADES, the robust accuracy increase while the clean accuracy drops. It is well known that adversarial training methods have a trade-off between clean and robust accuracy

As you say, SOVR sacrifices clean accuracies for robustness and it is the well-known trade-off. Even so, we would respectfully like to point out that SOVR (not TSOVR) achieves the best trade-off with high robustness in most cases if we define the trade-off as the summation of robust and clean accuracies as in the below table (the same as tables in General response).

**Table C Trade-off: Robust Acc. + Clean Acc.**
| |TRADES|TSOVR|SOVR|
|-|-|-|-|
|C10 (RN) | **131.3 $\pm$ 1** | 131.2 $\pm$ 0.2 | **131.3 $\pm$ 0.3**|
|C10 (WRN) |137.1 $\pm$ 0.7|136.4 $\pm$ 0.2 |**138.1 $\pm$ 0.2**|
|C10 (WRN)+AWP |140.5 $\pm$ 0.6|141.3 $\pm$ 0.4|  **142.9 $\pm$ 0.2**|
|SVHN |138.1 $\pm$ 1|138.4 $\pm$ 0.5|**138.5 $\pm$ 1**|
|C100| **77.0 $\pm$ 0.3** | 76.2 $\pm$ 0.2|76.4 $\pm$ 0.8|


Fig. 17 in Appendix F.6 shows that SOVR and TSOVR achieves the Pareto optimum under each condition: without regularization, with AWP, and with SEAT. We believe this result is not trivial because TRADES is known to be one of the best objective functions for adversarial robustness.

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Response to Reviewer G3JG
comment: We all thank you for your comments based on your expertise and your work on our paper. Following your comments, we revised the section of related work to clarify our contributions. We are afraid that we might not grasp your last concern well, and we would be grateful if you could add a little more information during the discussion.

> The idea is incremental because margin-based approaches and logit margin loss are natural and have been well studied before. 

To the best of our knowledge, our study is the first work that improves practical adversarial robustness by increasing logit margins through the principled way (Theorem 4.5). In addition, we could not find the study that revealed the difference of logit margin losses between AT and importance-aware methods. As you know, while several studies employ margin-based approaches, we could not find methods that improve logit margins theoretically and robustness against Auto-Attack at the same time. For example, (Tsuzuku et al. (2018) and Zhang et al. (2021a)) increase margins by considering Lipshitz constants theoretically. They derived the certified robustness, but their improvements of the practical robustness tend to be lower than adversarial training: Their robust accuracies on PGD are not better than AT. For practical robustness, MMA is the most successful margin-based method, and SOVR outperforms it. We would greatly appreciate it if you would inform us of other promising practical margin-based approaches.

> Additionally, it is also well-known that OVR approach helps to boost the margin.

Having read your comments, we rechecked the literature and could not find the studies that investigate the effect of OVR in logit margins of deep models, like our paper. In the research area of deep learning, we found [a] as a study using OVR. [a] used OVR to control the confidence on OOD data by using the difference of landscapes in data points. Whereas [a] does not provide any theoretical claims about logit margins, we theoretically reveal that OVR with gradient descent increases logit margins twice as large as cross-entropy in Section 4.  We added the discussion about [a] in Section 5.

In the context of SVM, OVR is well studied. Compared to SVM, our approach focuses on margins in the logit space by using gradient descent of neural networks whereas SVM focuses on margins in the data space.

[b] Padhy, Shreyas, et al. "Revisiting one-vs-all classifiers for predictive uncertainty and out-of-distribution detection in neural networks." ICML Workshop on Uncertainty and Robustness in Deep Learning 2020


> The proposed method only slightly outperforms the baselines.

We observed the statistical significance of SOVR by using the t-test with p-value of 0.05.

> The assumption in Eq. (4) is extremely infeasible and simple compared to real-world uses of the CE and logit margin losses.

Equation (4) is weighted cross-entropy and the exact objective functions of previous studies (GAIRAT, MAIL, and EWAT). Additionally, Eq.(4) includes usual cross-entropy (CE): If we choose $\bar{w}_n=1/N$ for all $N$ data points, Eq. (4) becomes CE. So, the analysis in Sec. 4.2 is valid for CE (we can set $w_2=1$ for cross-entropy). We would be grateful if you would explain your concerns about Eq. (4) in a little more detail.

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Response to Reviewer HRBR Part 1/2
comment: We thank you for your helpful comments based on your in-depth understanding of our paper. Following your comments, we revised our paper to clarify the motivation of $\log(1+e^{-z})$ and added the evaluation using logit scaling attacks, and other corrections in the revision. If you come up with other questions during the discussion, we would be glad to hear them.

> For Eq. (8), why did you choose $\log(1+e^{-z})$? The author should explain the rationale behind the choice. 

This is because $\log(1+e^{-z})=\mathrm{softplus}(-z)$ is numerically stable in training among the common functions that satisfy infinite sample consistency (ISC). Roughly speaking, 
ISC is a condition where the minimizer of a surrogate loss function (e.g., cross-entropy or OVR) minimizes the classification error, i.e., 0-1 error (Appendix D.1 in the revision).
It is known that ISC is satisfied when $\phi$ is a differentiable non-negative convex function and satisfies $\phi(z) < \phi(-z)$ for $z > 0$ (Zhang, 2004). 
Based on this, we tested $e^{-z}$ and $\log(1+e^{-z})$ because they satisfy the condition among common functions.

When using $e^{-z}$, we observed that overflow occurred in training. 
On the other hand, $\log(1+e^{-z})$ is numerically stable.
This might be due to the following two reasons:
First, it asymptotically approaches to $\max(-z,0)$, and overflow hardly occurs in the computation of $z$. 
Second, models can be regarded as a set of binary classifiers when using $\log(1+e^{-z})$.
Let the conditional probability for the class $k$ given $\mathbf{x}$ be 
$p(k|\mathbf{x})=\frac{1}{1+e^{-z_k(\mathbf{x})}}$.
We have $\ell_{OVR}(\mathbf{z}(\mathbf{x},\mathbf{\theta}),y)=\log (1+e^{-z_{y}(\mathbf{x})})+\sum_{k\neq y}\log(1+ e^{z_{k}(\mathbf{x})})
=- \log p(y|\mathbf{x})+\sum_{k\neq y}-\log(1-p(k|\mathbf{x}))$.
This is a negative log-likelihood for binary classification, 
and we can regard models as $K$ independent binary classifiers [a]. We add this discussion (Appendix D in the revision) and reference of [a]. 

[a] Padhy, Shreyas, et al. "Revisiting one-vs-all classifiers for predictive uncertainty and out-of-distribution detection in neural networks." ICML Workshop on Uncertainty and Robustness in Deep Learning 2020

> The authors say: “SOVR achieved better trade-off: SOVR achieved similar robust accuracies to TRADES, while it achieved better clean accuracy” However, the clean acc are 84.2 for TRADES and 83.1 for TSOVR. In addition, I disagree with the “better trade-off” for this table.

We are sorry for the unclearness of the statement and hope that the below explanation helps to clarify our claims. In the statement "SOVR achieved better trade-off...", we compared TRADES with SOVR but not TSOVR. The following tables summarize the results reported in Tables 1 and 2.

**Table D WRN34-10 without AWP**
| |AT|SOVR|TRADES|TSOVR|
|-|-|-|-|-|
|1. Robust Acc. |51.9 $\pm$ 0.5|53.1 $\pm$ 0.2|52.9 $\pm$ 0.3|**53.3 $\pm$ 0.1**|
|2. Clean Acc. |**85.6 $\pm$ 0.1**|85.0 $\pm$ 0.2|84.2 $\pm$ 0.7|83.1 $\pm$ 0.2|
|Trade-off (1+2) |137.5 $\pm$ 0.5|**138.1 $\pm$ 0.2**|137.1 $\pm$ 0.7|136.4 $\pm$ 0.2|

**Table E WRN34-10 with AWP**
| |AT|SOVR|TRADES|TSOVR|
|-|-|-|-|-|
|1. Robust Acc. |55.0 $\pm$ 0.2|56.03 $\pm$ 0.07|55.6 $\pm$ 0.4|**56.4 $\pm$ 0.2**|
|2. Clean Acc. |**87.8 $\pm$ 0.1**|86.9 $\pm$ 0.4|84.9 $\pm$ 0.5|84.9 $\pm$ 0.4|
|Trade-off (1+2) |142.8 $\pm$ 0.2|**142.9 $\pm$ 0.4**|140.5 $\pm$ 0.6|141.3 $\pm$ 0.4|

If we define the trade-off as the summation of robust and clean accuracies, SOVR achieves the best trade-off with higher or comparable robustness to TRADES. 
On the other hand, while the trade-off of TSOVR is not the best, TSOVR achieves the best robust accuracy. In addition, Fig. 17 in Appendix F.6 shows that SOVR and TSOVR achieves the Pareto optimum under each condition: without regularization, with AWP, and with SEAT. 

To clarify our claims, we modified the statement as "Among AT, SOVR, TRADES, and TSOVR, SOVR achieved the best trade-off:" in the revision.

> It is a contradiction to the author’s maintaining clean accuracy claim. 

We are afraid that the above statement “SOVR achieved better trade-off..." misled you into thinking that we claimed that SOVR maintains clean accuracy compared with cross-entropy. We admitted the drops of clean accuracies by the statement "SOVR slightly sacrifices clean accuracies under some settings" at the last of Sec. 6.2. This trade-off might be natural in the adversarial robustness as Reviewer aV6p pointed out. We believe the trade-off of SOVR (not TSOVR) is better than TRADES as discussed above.

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: Response to Reviewer HRBR Part 2/2
comment: > What are the results if the proposed method encounters the logit scaling attack (Hitaj)? In addition, it is better if the author can show some potential attack ideas that can break the proposed defense.

Thank you for the interesting question. 
We additionally evaluated SOVR by using logit scaling attacks and found that SOVR is not necessarily vulnerable to logit scaling while it is a bit less robust against PGD, on which logit scaling attacks are based, than other methods. 
We add the evaluation of logit scaling attacks with $\alpha=[0.1,1.0,10,100]$ (Fig. 18 of Appendix F.8 of the revision).
The result shows that SOVR can be the lowest robust accuracy (acc) against logit scaling attacks on CIFAR10 (RN18) but not under the other settings. However, since the robust acc of SOVR is not correlated to $\alpha$, this is caused by high robustness against PGD (Table 5) of previous methods rather than vulnerability to logit scaling of SOVR. Previous methods tend to be designed to increase robustness agasint PGD since Auto-Attack is a relatively recent attack. On the other hand, SOVR is designed to increase the robustness against the worst-case attack, and logit scaling attack is not the worst-case attack, which is Auto-Attack for now.

To produce attacks specialized for SOVR is a difficult problem since our method is designed to improve the robustness based on Definition 3.1 and Proposition 3.2. Since SOVR does not consider Lipschitz constants, a potential attack might be the attack that changes the logits the most, but it is a common threat for other defense methods. 

> Comparing Fig. 2(b) and Fig. 4, the peak shifting is a little weak. Is there any direction to improve this?

To shift the peak of logit margins more, we can increase $(M, \lambda)$ since $M$ is the rate of data points applied OVR to and $\lambda$ corresponds to $w_1$ in Eq.(16). In fact, Figs. 5(a) and 15(a) show that large $(M, \lambda)$ increases average of logit margin $|\ell_{LM}|$. Even so, we do not use such large values because we also care overfitting by increasing them. Since these values involve overfitting as shown in Figs. 5(b) and 15(b), moderate peak shifting can be the best for generalized robustness. 

> Observing Fig. 6, it seems that the proposed method did not work well if the number of classes increased. (Cifar-100 is much worse than Cifar-10.) It is better if the authors discuss this. Also, this is a potential limitation of the proposed method.

Thank you for the comments.
As you say, when the number of classes increases, the performance of AT tends to get close to that of SOVR. This is a piece of evidence that Theorem 4.5 explains the difference of logit margins between OVR and cross-entropy (CE). In certain finite time step $t$ (not the limit), Eqs. (16) and (17) show that the difference between OVR and CE depends on the number of classes $K$: Logit margins of OVR is about $2\log w_1 t$ and that of CE is about $\log Kw_2 t$.

Even so, Fig. 14 (b) shows that the increase rate of logit margins of OVR is larger than that of CE against epochs.
To achieve better performance, we can tune hyper-parameter $\lambda$, which corresponds to $w_1$ in Eq. (16) of Theorem 4.5.
When using $\lambda$= 0.6, we have the following results on CIFAR100. 

**Table F Tuning $\lambda$ on CIFAR100**
| |AT|SOVR original: $(\lambda)$=(0.5)|SOVR (0.6)|
|-|-|-|-|
|Robust Acc. |23.7 $\pm$ 0.3|24.3 $\pm$ 0.2|**24.6 $\pm$ 0.1**|
|Clean Acc. |**53.0 $\pm$ 0.7**|52.1 $\pm$ 0.8|51.9 $\pm$ 0.6|

After reading the above, you might think that CE also has the weight $w_2$ in Eq. (17). $w_2$ has been automatically tuned in GAIRAT, MAIL, and EWAT, and this tuning does not achieve comparable performance to SOVR. We are sorry that we did not thoroughly tune hyper-parameters on CIFAR100 at the submission time.

We added this discussion in Appendix F.10 of the revision 
and added the statement "We discuss the reason the rate of AT gets close to SOVR on CIFAR100 in Appendix F.10." at the last of Section 6.2.1 of the revision.

We thank you for several constructive comments again and are glad to improve the quality of our paper.

From: ICLR.cc/2023/Conference/Paper2328/Authors
title: General response 
comment: We greatly appreciate your work on the review process and are grateful for sparing time to write several constructive comments. 
On the basis of these comments, we revised our paper and submit the revision. **In our response, referred numbers (e.g., Section numbers and equation numbers) are based on the revision paper**.
We summarize the revised points and answer common questions here.
## Main revised points 
We add the following discussions on the basis of the comments in the revision:
- Justification of using $\phi=\log (1+e^{-z})$ in OVR (Appendix D)
- Comparison SOVR and TSOVR with TRADES (Appendix F.9)
- Histograms of the models obtained by early stopping in Figs. 2 and 4

To save space, we have moved the loss function of MART from Section 2 to the Appendix C.1.

## Reply to common questions
Some questions are shared among the reviewers, and we respond to them here. We provide detailed discussions in reply to each reviewer.
### Comparison with TRADES 
Since Reviewers aV6p and Fx8Y have a concern that the comparison to TRADES is lacked under some conditions, we evaluated TRADES and TSOVR on CIFAR10 (RN18), CIFAR100, and SVHN. We list the results as follows:

**Table A Robust Acc. against Auto-Attack**
| |TRADES|TSOVR|SOVR|
|-|-|-|-|
|C10 (RN) |48.8 $\pm$ 0.3|**49.8 $\pm$ 0.1**|49.4 $\pm$ 0.3|
|C10 (WRN) |52.9 $\pm$ 0.3|**53.3 $\pm$ 0.1**|53.1 $\pm$ 0.2|
|C10 (WRN)+AWP |55.6 $\pm$ 0.4|**56.4 $\pm$ 0.2**|56.03 $\pm$ 0.07|
|SVHN |49.4 $\pm$ 0.3|**49.87 $\pm$ 0.02**|48.5 $\pm$ 0.4|
|C100 |23.2 $\pm$ 0.1|**24.61 $\pm$ 0.07**|24.3 $\pm$ 0.2|

**Table B Clean Acc.**
| |TRADES|TSOVR|SOVR|
|-|-|-|-|
|C10 (RN) |**82.5 $\pm$ 1**|81.4 $\pm$ 0.2|81.9 $\pm$ 0.2|
|C10 (WRN) |84.2 $\pm$ 0.7|83.1 $\pm$ 0.2|**85.0 $\pm$ 0.2**|
|C10 (WRN)+AWP |84.9 $\pm$ 0.5|84.9 $\pm$ 0.4|**86.9 $\pm$ 0.24**|
|SVHN |88.7 $\pm$ 1|88.5 $\pm$ 0.5|**90.0 $\pm$ 1**|
|C100 |**53.8 $\pm$ 0.3**|51.6 $\pm$ 0.2|52.1 $\pm$ 0.8|

In these tables, TSOVR achieved the best robust accuracy against AutoAttack, and SOVR achieved better robust accuracy than TRADES under some conditions.
We did not evaluate SEAT + TRADES since (Wang & Wang, 2022) reported that TRADES degrades the performance of SEAT.
### Sacrificing clean accuracy
Reviewers HRBR and aV6p have a concern that SOVR drops clean accuracy while Reviewer aV6p admits that it is a well-known trade-off.
Having read these comments, we evaluated the trade-off by summing robust and clean accuracies as follows:

**Table C Trade-off: Robust Acc. + Clean Acc.**
| |TRADES|TSOVR|SOVR|
|-|-|-|-|
|C10 (RN) | **131.3 $\pm$ 1** | 131.2 $\pm$ 0.2 | **131.3 $\pm$ 0.3**|
|C10 (WRN) |137.1 $\pm$ 0.7|136.4 $\pm$ 0.2 |**138.1 $\pm$ 0.2**|
|C10 (WRN)+AWP |140.5 $\pm$ 0.6|141.3 $\pm$ 0.4|  **142.9 $\pm$ 0.2**|
|SVHN |138.1 $\pm$ 1|138.4 $\pm$ 0.5|**138.5 $\pm$ 1**|
|C100| **77.0 $\pm$ 0.3** | 76.2 $\pm$ 0.2|76.4 $\pm$ 0.8|

If we use this definition of the trade-off, SOVR achieves the best in most cases. In addition, Fig. 17 in Appendix F.6 shows that SOVR and TSOVR achieve the Pareto optimum under each condition; without regularization, with AWP, and with SEAT. 

We would like to thank you again. If any concerns arise or remain, we would be happy to hear from you.

From: ICLR.cc/2023/Conference/Paper2328/Reviewer_HRBR
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a novel adversarial training approach, which can increase the logit margin without losing too much clean accuracy. The authors discover that the logit margin of traditional adversarial training has two peaks. Based on this, they designed a switching mechanism and a novel margin loss function (OVR) to improve traditional adversarial training, aiming to push the right peak to the left while maintaining the left peak. Experiments show superiority over baselines.
strength_and_weaknesses: Pros:
- The paper is well written and easy to follow: start from some observations, assumptions, theorems, and the design of their approach.
- The finding about logit margin distribution is interesting. The corresponding analysis is also solid.
- The experiments demonstrate the effectiveness of the proposed method.

Cons:
- For Eq. 8, why did you choose log(1+e^-z)? The author should explain the rationale behind the choice.
- In Table 2, if we add SOVR to Trades (TSOVR), the clean accuracy decreases. It is a contradiction to the author’s maintaining clean accuracy claim. AWP results show the same phenomenon, i.e., clean accuracy decreases while applying the proposed method. 
- The authors say: “SOVR achieved better trade-off: SOVR achieved similar robust accuracies to TRADES, while it achieved better clean accuracy” However, the clean acc are 84.2 for TRADES and 83.1 for TSOVR. In addition, I disagree with the “better trade-off” for this table.
- What are the results if the proposed method encounters the logit scaling attack (Hitaj)? In addition, it is better if the author can show some potential attack ideas that can break the proposed defense. 
- Comparing Fig. 2(b) and Fig. 4, the peak shifting is a little weak. Is there any direction to improve this?
- Observing Fig. 6, it seems that the proposed method did not work well if the number of classes increased. (Cifar-100 is much worse than Cifar-10.) It is better if the authors discuss this. Also, this is a potential limitation of the proposed method.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written, and the overall quality is good. The observation is novel. All the experiment details are in the appendix.


summary_of_the_review: The observation in the paper is interesting, but the experiment results are not convincing to support the claims.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2328/Reviewer_G3JG
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper first studies the relationship of logit margin loss $\ell_{LM}$ and adversarial robustness and then proposes using one-versus-the-rest (OVR) loss to improve the logit margin loss.
strength_and_weaknesses: Strength
- The paper is well-written.
- The behavior of the logit margin loss and how OVR loss helps to improve logit margin loss is well-studied.

Weaknesses
- The idea is incremental because margin-based approaches and logit margin loss are natural and have been well studied before. Additionally, it is also well-known that OVR approach helps to boost the margin.
- The proposed method only slightly outperforms the baselines.
- The assumption in Eq. (4) is extremely infeasible and simple compared to real-world uses of the CE and logit margin losses.

clarity,_quality,_novelty_and_reproducibility: Although the paper is well-written and the concepts are deliberately empirically studied, the novelty of this idea is incremental.   The experimental results show that the proposed approach slightly improves the baselines.
summary_of_the_review: The paper is well-written. The concepts are deliberately empirically studied. However, in my opinion, the idea is incremental.

-----------------------------------------------------------------------------------------------
Thanks the authors for addressing my concerns. I still believe that the idea is incremental and the theory needs to make strong assumptions. Therefore, I am keen on my current score.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: There are no ethics concerns.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2328/Reviewer_aV6p
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work reveals that the adversarial examples used in adversarial training can be roughly divided into difficult examples and easy examples according to their logit margins. The proposed SOVR method aims to reduce the adversarial examples which have larger logit margins, then enhance the robustness of DNN models. 
strength_and_weaknesses: I'm confused about "On the other hand, logit margins of importance-aware methods concentrate near zero, and thus, importance-aware methods reduce the logit margins of easy samples. This implies that the weighted cross-entropy used in importance-aware methods is not very effective in increasing logit margins. To increase the logit margins of difficult samples, ...". First, why the observation of "importance-aware methods reduce the logit margins of easy samples" can derive "importance-aware methods is not effective in increasing logit margins"? Second, why do you want to increase the logit margins of difficult samples? Do the large logit margins indicate the examples are well learned? And the proposed method is to minimize the logit margins of "easy samples" if I didn't misunderstand.

And some questions about experiments are as follows:
1. What about TRADES perform in Figure 2? Can the histogram of TRADES also be divided into two peaks?
2. In Table 1, what about the results of TRADES?
3. In Table 2, I'm afraid that the results can't convince me that the proposed method can increase the robustness. Compare with AT+AWP and TRADES, the robust accuracy increase while the clean accuracy drops. It is well known that adversarial training methods have a trade-off between clean and robust accuracy. The performance increase of compare with SEAT is marginal.
clarity,_quality,_novelty_and_reproducibility: The clarity needs to be improved.
summary_of_the_review: Confusing description and lack of convincing empirical results.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2328/Reviewer_Fx8Y
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The paper studies the source of the worse performance (i.e. lower adversarial robustness) of importance-aware methods compared to standard adversarial training (and variations like TRADES). It is argued that importance-aware methods yield small logits margins for all points, while naive adversarial training shows large margins for a considerable portion of training points (easy points). This has the effect of reducing robustness on the easy examples. This motivates SOVR, which uses the cross-entropy loss on the easy points and the novel OVR loss on the difficult ones. In the experimental evaluation, SOVR outperforms importance-aware methods, and can be combined with regularization methods like TRADES and AWP.
strength_and_weaknesses: Strengths
- The analysis of the logits margins is interesting and suggests an explanation for the worse performance of importance-aware methods compared to standard adversarial training.

- The proposed SOVR is a reasonable solution in light of the analysis of the behavior of the logits margins, and in the experimental evaluation is shown to be effective.

Weaknesses
- The histograms of the logits margin shown in the paper are mostly for model at the last epoch of training. In Fig. 8 in the appendix one can see that, when using the model selected by early stopping, the left peak for adversarial training is much less pronounced, suggesting that it is mainly due to overfitting. While there's still a difference in the distribution of the margins between naive adversarial training and the importance-aware methods, I think the plots for the models with early stopping should be reported in the main part to give a more complete picture of the phenomenon the paper analyzes.

- While SOVR outperforms importance-aware methods, a comparison to other methods (see e.g. https://robustbench.github.io/) for obtaining robust models is missing. The only direct comparison to TRADES is with WRN on CIFAR-10, and the two methods achieve similar results. SOVR getting similar or better results than existing methods would significantly strengthen the paper.
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well written, and the results clearly presented.

Quality, Novelty: the analysis is interesting and the proposed SOVR shows good performance. Both are, to my knowledge, novel.

Reproducibility: sufficient details are provided.
summary_of_the_review: The paper presents an interesting analysis of importance-aware methods and a new training scheme to improve their results. A more thorough comparison to existing methods for robust classifiers would increase its relevance.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold