From: auai.org/UAI/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept (Poster)
comment: Meta Review: All reviewers had a (mildly) positive view about the paper, so acceptance is merited.

From: auai.org/UAI/2022/Conference/Paper45/Reviewer_Rh4m
Q1 Summary and contributions: The paper studies Test Time Augmentation for improving test time performance. In particular, it proposes to use cyclic search for suitable transformations as well as an entropy weight method. The experimental result shows that it improves standard image classification as well as robustness to image corruptions.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 2: Fair: The paper is likely to have moderate impact within a subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: + The usage of cyclic search for suitable transformations seems novel and interesting.

+ The experiment results are good compared to existing TTA methods

+ The overall writing is clear and easy to follow
Q4 Main weakness: - There's a missing line of related research topic, which is test-time training. How does the proposed method (TTA) perform compared to TTT-based methods?

- The evaluation is not that thorough. Though both standard image classification as well as robustness to image corruptions are tested, only one dataset is used, which might cause the problem of whether it generalizes to other datasets.

- What is the inference computation cost of Cyclic method? How does it compare to standard inference, as well as other TTA methods?
Q5 Detailed comments to the authors: Please refer to the previous sections.
Q6 Overall score: 5: Borderline accept: Technically solid paper where reasons to accept, e.g., novelty, outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Q7 Justification for your score: I think in general the paper has merits. I would like to see other reviews and the author response on my questions.
Q8 Confidence in your score: 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper45/Reviewer_c8J3
Q1 Summary and contributions: This paper investigates a type of data argumentation method, called test time augmentation. By introducing cyclic optimization process and entropy weight method, their proposed EWM can improve the instance-level TTA with a more flexible magnitude, and achieve better ability. Some experiments are conducted to verify the advantage of their approach.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 2: Fair: The paper is likely to have moderate impact within a subfield of AI.
Q2(3) Correctness/Technical quality: 2: Fair: The paper has minor, easily fixable, technical flaws that do not impact the validity of the main results.
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: + Sound technical contribution. They assemble powerful techniques to improve the ability of  test time augmentation. The introduce of each   technique is reasonable and useful.
+ Good presentation. It is good to follow. 
Q4 Main weakness: - It is somehow trivial to achieve better performance because some advanced techniques are beneficial to proposed approach.
- Ablation study is needed to verify which of cyclic learning or entropy weight method is more significant.
- More experiments on other datasets are also needed.
Q5 Detailed comments to the authors: The interaction of cyclic learning or entropy weight method should be discussed.
And more experiments are needed to verify proposed method, including more datasets, more ablation, more baselines.
Q6 Overall score: 5: Borderline accept: Technically solid paper where reasons to accept, e.g., novelty, outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Q7 Justification for your score: The techniques are sound, but experiments are inadequate.
Q8 Confidence in your score: 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper45/Reviewer_qnwN
Q1 Summary and contributions: The paper proposes cyclical test-time data augmentation (TTA) --- an adaptive method that iteratively selects a sequence of optimal augmentations that will lead to the best loss according to a loss predictor network. The authors also propose another idea ---  Entropy Weight Method (EWM) for weighting the predictions under different augmentations for the case when several alternative augmentations are selected for each image in TTA.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 2: Fair: The paper contributes some new ideas.
Q2(2) Significance/Impact: 2: Fair: The paper is likely to have moderate impact within a subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 2: Fair: The paper is somewhat clear, but some important details are missing or unclear.
Q3 Main strengths: - The proposed idea for cyclical TTA is very logical
- The proposed method is simple, and computationally not very expensive
- Some of the results are good, particularly mCE for a Standard ResNet-50
Q4 Main weakness: - The writing is often unclear
- The motivation for the EWM is not obvious
- The results on the Augmix-trained network don't look particularly good
- Generally, the empirical evaluation does not convincingly show that the method is useful in practice
Q5 Detailed comments to the authors: The paper proposes a very simple extension of [1], where instead of just one augmentation, a sequence of augmentations is picked according to the loss predictor. Several questions:
- How was the loss predictor network trained in the experiments? Is it trained using the corruptions from ImageNet-C or just the original train data?
- How many augmentations do your cyclical policies end up inclluding?
- Can you apply the method of [1] to sequences of augmentations, e.g. consider all the possible augmentation sequences of length 5? Would that be feasible computationally? I imagine, it should work at least as well as the proposed method.

The propose EWM weighting method is not motivated very well in my opinion. In particular, regular averaging of the softmax class probabilities is already taking uncertainty into account. Indeed, imagine  for one of the augmentations the network is 99% confident in class 0, and for the second augmentation it's 50% confident in class 1. Then, the prediction will be class 0. So why do you think we need an extra confidence-based weight for the different augmentations?

The empirical results look good on the standard network, where the cyclical TTA with both averaging types significantly outperforms the standard fixed augmentation baselines. However, on the Augmix-trained network the results are not as good. In particular, 5 crops and 10 crops fixed augmentation methods perform better than the proposed adaptive cyclical TTA. Consequently, the empirical results are mixed.
- Why do you think the performance on Augmix-trained network is not good?
- Are there other problems that could be interesting to consider in order to show that the proposed method works well? Perhaps, you could consider CIFAR datasets, e.g. CIFAR-100 with a WideResnet and ResNext, following the setting considered in [1]?

Finally, the writing is not good, often confusing. The structure of the paper is reasonable, but the phrasing of the sentences is often hard to parse. Just as an example:
> In other words, the loss predictor tells of what transformation would result in the best outcome for the target network, in scores of minimum the better.
I don't get what it means, and a lot of sentences in the paper are like this. I ask the authors to carefully go through the paper and try to make sure that all the sentences are clearly phrased. 

**References**

**[1]** *Learning Loss for Test-Time Augmentation*
Ildoo Kim, Younghoon Kim, Sungwoong Kim
Q6 Overall score: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
Q7 Justification for your score: The paper proposes a very simple and logical idea, which I believe could be highly practical. However, the empirical evaluation is not entirely convincing, as the method works well for one model, and not as well for the other. 
Q8 Confidence in your score: 4: Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation.
Q9 Complying with reviewing instructions: 1: Yes.