Paper: AUGCAL: Improving Sim2Real Adaptation by Uncertainty Calibration on Augmented Synthetic Images (ICLR-2024)

Reviewer comment (Evaluation):
The choice of augmentation should satisfy the property 1 in Sec. 3.2.2. Even though Tab.1 shows that PASTA/R.Aug-SIM are closer to Real than SIM based on MMD, this is not intuitive. It would be better to clarify it more.

Author response:
**PASTA vs RandAug distributional distances**

PASTA was designed specifically to tackle Sim2Real discrepancies by introducing increased perturbations to high-frequency synthetic image components. RandAug, on the other hand, has no such constraints, and can generally perturb all (low, high frequency) components indirectly using chained photometric operations. From the measured distributional distances in Table 1 of main paper, PASTA and RandAug are similarly effective in reducing distributional distance on GTAV$\to$City, whereas RandAug is more effective than PASTA on VisDA-C. We believe this is due to the nature of VisDA-C synthetic images. Synthetic images in VisDA-C are snapshots of CAD models of object classes from different viewpoints, lighting conditions in clear backgrounds. Consequently, increased focus towards high-frequency components (PASTA) in such settings could be less beneficial compared to peturbing all frequency components (RandAug). Using both AUG choices (Sim2Real specific and otherwise) allowed us to better demonstrate the general applicability of AUGCAL for Sim2Real settings.

Reviewer comment (Evaluation):
For PASTA, it is designed to bridge the syn-to-real gap and maybe reduce the artifacts of synthetic data, but it is unclear why R.Aug is even closer to Real.

Author response:
**PASTA vs RandAug distributional distances**

PASTA was designed specifically to tackle Sim2Real discrepancies by introducing increased perturbations to high-frequency synthetic image components. RandAug, on the other hand, has no such constraints, and can generally perturb all (low, high frequency) components indirectly using chained photometric operations. From the measured distributional distances in Table 1 of main paper, PASTA and RandAug are similarly effective in reducing distributional distance on GTAV$\to$City, whereas RandAug is more effective than PASTA on VisDA-C. We believe this is due to the nature of VisDA-C synthetic images. Synthetic images in VisDA-C are snapshots of CAD models of object classes from different viewpoints, lighting conditions in clear backgrounds. Consequently, increased focus towards high-frequency components (PASTA) in such settings could be less beneficial compared to peturbing all frequency components (RandAug). Using both AUG choices (Sim2Real specific and otherwise) allowed us to better demonstrate the general applicability of AUGCAL for Sim2Real settings.

Reviewer comment (Experiments):
If large-scale unlabeled real-world data is available, it is worth considering whether style transfer-based augmentation would have better performance because it directly uses unlabeled real-world data.

Author response:
**Would style-transfer operations be eligible AUG choices?**

Good point! While a style-transfer operation that can translate synthetic images to real style (while preserving semantics) can help reduce distributional distance, it cannot perfectly mitigate the distributonal gap. This is because Sim2Real discrepancies exist not only in terms of appearance, but also content. For instance, compared to Cityscapes, synthetic scenes from GTAV tend to be much less populated with vehicle and human instances. Similarly, synthetic source images in VisDA-C contain clear backgrounds (no surrounding context) unlike the real target counterparts. 

Further, combining deep network (DNN) based style-transfer operations with state-of-the-art Sim2Real UDA methods is non-trivial since -- (1) it necessitates pre-training a translation network for every Sim2Real shift, (2) running DNN based style transfer online in conjunction with UDA is expensive and (3) doing such translations for synthetic source images offline would require quality checks for different Sim2Real shifts. Our intent behind experimenting with PASTA and RandAug as Aug transforms in AUGCAL was to consider simple eligible transforms (based on properties 1 and 2) that are inexpensive to combine with Sim2Real UDA methods while having shown demonstrable benefits in such settings (as shown in [A]).

[A] - PASTA: Proportional amplitude spectrum training augmentation for syn-to-real domain generalization, ICCV 2023.

Reviewer comment (Evaluation):
Since property 1 is a distribution distance, it is unclear how those appearance augmentations affected the distribution.

Author response:
**How do appearance augmentations affect marginal distributions?**

Thanks for bringing this up. Property 1 is based on distributional distances of the marginal input (image) distributions. Assuming covariate shift conditions, we parameterize this (and measure in Table 1) in terms of image feature distances. Since appearance augmentations on images impact features, they in turn affect feature distances or equivalently distributional distances.

Reviewer comment (Evaluation):
Usually adding calibration loss will have lower ECE but also lower accuracy (in your case mIoU). Can you give some intuition in why adding calibration loss on source give better mIoU as shown in Table 3 (a)?

Author response:
Good catch! For in-distribution settings, an additional training time calibration loss ($\mathcal{L}\_{\text{CAL}}$) doesn't necessarily compromise accuracy for reduced ECE (as shown for DCA, MDCA, MbLS). In our Sim2Real UDA setups, especially for methods that rely on self-training (or entropy minimization), errors on real data often stem from overconfident predictions. We believe adding only a training time calibration loss ($\mathcal{L}\_{\text{CAL}}$) can help reduce overconfidence, leading to better pseudo-labels during adaptation, which can *potentially* improve adaptation performance in some cases. However, this behavior of minor improvements with only CAL is not universally applicable across settings, unlike AUGCAL. Do let us know if this helps answer your question.

Reviewer comment (Novelty):
The technical contribution of this paper is a bit limited. The key components of AUGCAL are the data augmentation and DCA based calibration loss. However the data augmentation technique is commonly used in domain adaptation and the DCA proposed by previous work for model calibration.

Author response:
**Limited Technical Contributions**

We would like to clarify our key contributions. The intent behind our proposed method, AUGCAL, was to reduce miscalibration of Sim2Real adaptation methods while preserving transfer performance. We first make the observation that errors by Sim2Real adaptation methods on "real" data are tied with increasing overconfidence on mispredictions. Based on the established goal and our observations,

- We first provide a simple analytical justification surrounding how training time calibration interventions (CAL) on labeled "synthetic" source data can be helpful in reducing miscalibration on "real" target data. Building on top of this, we further demonstrate how incorporating **eligible** AUG transformations in this vanilla CAL setup (to form AUGCAL), can be even more effective in reducing Sim2Real miscalibration.
- We empirically validate the effectiveness of AUGCAL by showing that it reduces Sim2Real miscalibration while retaining performance across multiple tasks, adaptation methods, intra-task backbones (CNNs & Transformers), and shifts. Additionally, we also assess AUGCAL effectiveness across multiple AUG (PASTA, RandAug) and CAL (DCA, MDCA, MbLS) choices.

We would like to re-emphasize that the key technical contribution in AUGCAL lies precisely in demonstrating that a combination of eligible (in accordance with the properties in Sec. 3.2.2) AUG transforms with training time CALibration losses is effective in reducing Sim2Real miscalibration across multiple settings while being easy to integrate (in a plug-and-play fashion) in existing setups.

Reviewer comment (Experiments):
The proposed method is only compared with diffident backbone UDA models. A more convincing comparison would be to compare with existing confidence calibration methods, such as [Wang et al. 2022], [Gong et al. 2021].

Author response:
**Limited Comparisons**

Thanks for the suggestion. In out-of-distribution settings, both [A,B] study "*post-hoc calibration methods*" (based on temperature-scaling) for "*zero-shot generalization*" (when target data is **unavailable**). In contrast, we study "*training time-calibration methods*" (calibration losses at training time) for "*unsupervised adaptation*" (when modestly sized unlabeled target data is **available**). We note that the distinction between the two experimental settings is crucial, since unlike zero-shot settings (key experiments in [A], Sec 6.1.2 results in [B]), adaptation scenarios consist of repeated model updates based on intermediate miscalibrated target predictions. This makes adaptation scenarios much more sensitive to progressive miscalibration on target data.

That said, in addition to demonstrating the effectiveness of AUGCAL across different Sim2Real adaptation models (across tasks, UDA methods, intra-task backbones, and shifts), we also conduct experiments to assess AUGCAL effectiveness for different CAL choices ([C, D, E]; which are effective confidence calibration techniques on their own). Given the limited timeframe (review received two days before end of discussion period), we are unable to perform a new equivalent comparison (by adapating [A,B] to our experimental setting).  We would like to instead point the reviewer to our temperature-scaling (TS) comparisons on VisDA-C (in Sec 5.2 main paper; Sec. 5 appendix) where we show that TS methods overfit substantially to improved calibration on "synthetic" data, making them ineffective for improving calibration on "real" data for Sim2Real adaptation methods.

[A] - Confidence Calibration for Domain Generalization under Covariate Shift (Gong et. al., 2021)

[B] - On Calibrating Semantic Segmentation Models: Analyses and An Algorithm (Wang et. al., 2022)

[C] - Improved trainable calibration method for neural networks on medical imaging classification, BMVC 2020

[D] - A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration, CVPR 2022

[E] - The devil is in the margin: Margin-based label smoothing for network calibration, CVPR 2022

Reviewer comment (Presentation):
The paper exceeds the page limit of 9 pages.

Author response:
**Manuscript Length**

The current revised submission (based on suggestions from R-dFNn, R-WuQd) is contained with 9 pages for the main text, with the reproducibility and ethics statement being on page 10. The author guide (see [F]) clearly specifies that both the ethics and reproducibility statements do not count towards the page limit.

[F] - https://iclr.cc/Conferences/2024/AuthorGuide