Paper: Calibration Attack: A Framework For Adversarial Attacks Targeting Calibration (ICLR-2024)

Reviewer comment (Presentation):
Questions about Figure 1. As mentioned in its caption, red bars represent accuracy and blue bars represent confidence. However, the vertical axis in your image is also labeled as accuracy. Could you explain this in detail? Besides, I noticed that Figure 1 is in PNG format. A vector graphic format might be better.

Author response:
Q1: The x-axis represents the confidence range, and the y-axis is the accuracy in that range of confidence scores.

Reviewer comment (Evaluation):
How to calculate the average confidence? In my opinion, letting "average confidence = the product of the sample proportions in each bin" would provide a reasonable explanation according to your setting. However, in Table 1, I think the average confidence is not calculated in this way. Otherwise, we could have Accuracy = ECE +/- average confidence.

Author response:
Q2: Average confidence in the tables is referring to the averaged predicted confidence score for all of the tested data. In the context of the ECE calculation it is the same but relative to a specific bin.

Reviewer comment (Novelty):
Relation with previous works. I list two more related works on robust calibration. It would be perfect if you could compare your work with theirs. [1] Tang Y C, Chen P Y, Ho T Y. Neural Clamping: Joint Input Perturbation and Temperature Scaling for Neural Network Calibration, arXiv:2209.11604. [2] Yu Y, Bates S, Ma Y, et al. Robust calibration with multi-domain temperature scaling, in NeurIPS 2022.

Author response:
Q3: Although it would be interesting to test these additional baselines, we have already tried many baseline defense methods and we cannot test every possible method of this type. We believe the current methods we tested are a good overview of the general performance.

Reviewer comment (Reproducibility):
The implementation of adversarial attack/defense algorithms. I found it very hard to parse the results of the experiments. I wonder if the authors could provide the corresponding code to the results in Table 1 (or at least a demonstration) so that I can reproduce and check the results.

Author response:
Q4: The code for the results in table 1 and the rest of the results will be provided later as we will put the full code on GitHub. For now, are there any specific questions about the implementation as we cover our algorithm and training details in detail in the paper?

Reviewer comment (Novelty):
My primary concern with the paper is around originality and its failure to cite prior work. The paper claims to be the first to study attacks on confidence scores, however a very similar test-time attack was proposed by Galil & El-Yaniv (2021). More recently a training-time attack on confidence scores was proposed by Zeng et al. (2023) (which first appeared on arXiv in 2022). It’s unfortunate these papers are not cited. There is also closely related work on certifying confidence scores by Kumar et al. (2020) and Emde et al. (2023) which ought to be cited.

Author response:
Weaknesses 1 and 2: We greatly appreciate making us aware of these other previous works that are in a similar vein to our paper, and we will acknowledge and reference them in our revision. Nevertheless, we still believe that our contributions over these works are substantial, as none of the prior works comprehensively study the range of the 4 attacks we introduce and cover in the context of calibration rather than uncertainty estimation, nor do they systematically investigate how well victim models would remain well calibrated under such attacks, including under a range of different calibration methods and adversarial defense methods, including a few novel ones like CAAT and CS. Unlike previous works, we examine in detail the effects of attacks targeting overconfidence as well, and evaluate the detectability relative to standard attacks both in terms of detectability methods and visualization approaches like Grad CAM.

Reviewer comment (Novelty):
In order to better assess originality, I have compared this paper with Galil & El-Yaniv (2021). As far as I can tell, the differences between the attacks are minor:
- Galil & El-Yaniv focus on reducing the confidence of the predicted class, whereas this paper also considers increasing the confidence.
- Galil & El-Yaniv’s attack algorithm is FGSM-based, whereas this paper uses the Square attack algorithm and PGD.
- Galil & El-Yaniv maximize/minimize the confidence of the predicted class directly, whereas this paper also considers the margin.  
Apart from attacks, this paper also contributes some insights on defences, which is not something that Galil & El-Yaniv cover. However, overall I don’t believe this paper’s contributions are sufficiently original/significant, at least in its current state.

Author response:
Weaknesses 1 and 2: We greatly appreciate making us aware of these other previous works that are in a similar vein to our paper, and we will acknowledge and reference them in our revision. Nevertheless, we still believe that our contributions over these works are substantial, as none of the prior works comprehensively study the range of the 4 attacks we introduce and cover in the context of calibration rather than uncertainty estimation, nor do they systematically investigate how well victim models would remain well calibrated under such attacks, including under a range of different calibration methods and adversarial defense methods, including a few novel ones like CAAT and CS. Unlike previous works, we examine in detail the effects of attacks targeting overconfidence as well, and evaluate the detectability relative to standard attacks both in terms of detectability methods and visualization approaches like Grad CAM.

Reviewer comment (Presentation):
Regarding the presentation of Section 5: I found it confusing that the calibration methods (TS, Splines, DCA, SAM) are presented alongside the defenses (AAA, AT, CA AT, CS). I think there is a risk that some readers may misinterpret the calibration methods as defenses, even though they are not designed to defend against attacks.

Author response:
Weakness 3: Indeed, they are not traditional defenses, but in the context of our work we are trying to examine whether by trying to better calibrate a model through traditional calibration methods leads to any sort of robustness against calibration attacks.

Reviewer comment (Evaluation):
The authors' motivation for calibration attack is that the prediction score instead of the classified label is directly used in downstream tasks. However, most of the experiment results are presented in terms of the calibration error, but did not explore further on its effect on the downstream. Without a concrete example, it is hard to assess the significance on the implication of this error on confidence score.

Author response:
Motivation: The effect on average classification confidence can be seen, including a t-sne diagram in the appendix showing the effects of the attacks on the decision boundary. Calibration error is also the primary means of determining how suitable a model’s predicted confidence scores are. We can refer to papers such as Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networks (Ragni et al.) where in an ASR system downstream confidence scores are used for flagging up words that need further review for not wasting resources annotating data that is too easy, in which case edge cases are identified through confidence scores. Both of these require low calibration error for example. In the both cases, overconfidence attacked data will make it impossible to do these tasks. We can add these discussions in our revision.

Reviewer comment (Experiments):
The attack algorithm is independent between samples, but is it possible to inferred information about the model on its tendency to be over/under-confident and reduces the query sizes?

Author response:
Q1: That is an interesting question. We thought about this previously, but determined that it does not make that big of a difference targeting a model’s existing weaknesses by estimating its distribution because the max and random attacks tend to be more effective in the long run. One can implement targeting the specific calibration distribution this naturally by choosing the corresponding one of the attack types if one wishes (e.g., if average predicted confidence is far higher than accuracy then use overconfidence attack). The 4 different attacks we create offer a lot of flexibility in terms of how you would want to attack a model, and on which. Many derived attacks can be created, if same one determines that at a confidence level of 70% the model is very inaccurate, hence we should use the underconfidence or overconfidence attack to place more incorrectly classified samples into that range. This could be more efficient if one is using very small query sizes, but in the long run the maximum attack will be more effective, and the random attack will completely result in scrambled confidence scores.

Reviewer comment (Experiments):
Is it still possible to perform the attack if only the predicted class label can be queried?

Author response:
Q2: Based on the current implementation, it is unlikely, though you could pick a black box attack that only makes use of predicted class labels and run the maximum attack on the samples based only the class labels and we believe the effect should be the same, although with less efficiency.

Reviewer comment (Experiments):
Is it possible to attack the training process and have the model be over/under-confident on unaltered samples?

Author response:
Q3: This is interesting but out of scope for this paper as we assume that classifiers are already trained and this scenario would be more of a data poisoning attack. In addition, existing models tend to lean towards overconfidence particularly when training on cross-entropy loss and that this is an inherent learning bias. We know that by feeding calibration attacked samples the inherent calibration of the model (on clean data) is made worse as it does with all of the adversarial training methods, but a sort of data poising approach we do not think it is as viable a scenario as we primarily want to introduce the issue in its most likely form.

Reviewer comment (Novelty):
It would benefit more readers if the authors put their emphasis on the significance of calibration attack, instead of the specific method. Since the paper is proposing a new problem, which is brave, the most important thing would be claiming that it is a worthwhile thing. I am not convinced by the only illustration of autonomous driving in the introduction. The design of each method looks too complicated and distractive.

Author response:
The autonomous driving example is only one such example that we put in because of the GPTSRB dataset. We can add and discuss an example in the medical domain where a doctor is using a model to determine whether x-ray scans contain cancer. In this case by attacking the scans and making the model miscalibrated without compromising the accuracy, edge cases that may or may not contain cancer can be made to be overconfident, meaning it is easier for the domain expert to ignore them. Vice versa, easy cases that contain no cancer can be made significantly underconfident, flooding the domain expert (doctor) with a lot of unnecessary work double checking scans. We will update this in our revision to make it clearer.

Reviewer comment (Novelty):
Response to rebuttal: Thanks for the rebuttal. I agree that this is a decent work, but the contribution is not significant enough for ICLR, as also mentioned by Reviewer YLZS. The efforts are far from sufficient to convince the community that the new problem is significant.

Author response:
It is true that the extension is not complicated, but many papers present techniques or methods that are simple modifications or variations of previous one, like the temperature scaling technique in the famous Guo et al. 2017 calibration paper, which was a simple modification of the previous Platt Scaling method, but provide strong results or present an interesting empirical analysis. We believe our paper fits into this category as no one has looked at adversarial attacks from the standpoint of attacking calibration without affecting the accuracy, and particularly examined aspects like efficiency, detection rate, defense method effectiveness etc. This is simply not an issue that the community is aware of, and without this paper vulnerable systems may not have the insights to see how these kinds of attacks can occur and comprise their performance. In particular studying attacking through overconfidence is not something that was really featured in previous works, and we believe that the results are interesting, including the insight of how it is more difficult to attack overconfidence than underconfidence.