Paper: Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Domain Adaptation (ICLR-2024)

Reviewer comment (Theory):
The method is heuristic and not clearly motivated. While it is known that a mixup training procedure yields more calibrated models, it is unclear why learning Temperature Scaling using a convex combination of pseudo-labeled samples yields meaningful temperature values.

Author response:
While we acknowledge that our method is heuristic, it's important to highlight that the designs of our PseudoCal method are rooted in clear motivations and can be intuitively explained. **The inspiration for the PseudoCal framework stems from the factorized negative log-likelihood (NLL) objective**. The factorization implies that if two datasets share the same correct-wrong statistics (i.e., an identical count of correct and wrong predictions), they should exhibit similar temperatures after NLL optimization. In the context of PseudoCal, our objective is to synthesize a labeled set that mirrors the correct-wrong statistics of real target samples. We choose to **employ mixup** for sample synthesis due to its simplicity and its **ability to elegantly leverage the target-domain data structure learned by the UDA model**. Additionally, **leveraging the well-established cluster assumption**, we anticipate a sample-level correspondence between mixed samples and real target samples, facilitated by suitable values of the mix ratio $\lambda$. Accurate sample-level correspondences ensure similar correct-wrong statistics, ultimately guaranteeing that the temperature obtained with the labeled pseudo-target set closely approximates the Oracle target-domain temperature.

Reviewer comment (Theory):
It is not explained how the proposed method is derived from the UDA situation.

Author response:
One novelty of our PseudoCal approach lies in adopting a **target-domain perspective** to address the calibration problem in UDA, distinguishing it from existing methods that primarily rely on source data to address the covariable shift challenge. It's essential to note that **in the UDA field, various studies, including many in the recently popular source-free UDA settings, also operate without source data.** 
Similar to previous target-specific works, PseudoCal **relies on a well-trained UDA model that effectively captures the target-domain data structure.** The absence of source data is compensated by leveraging the knowledge embedded in the UDA model. In situations where the UDA model struggles to learn the target-domain structure, the cluster assumption may not hold, potentially damaging the sample-level correspondence between mixed samples and real target samples. This aspect is evident in **Table 6, where, in the I$\to$S task, both SHOT and DINE models exhibit low accuracy, resulting in high calibration errors** even after being calibrated with PseudoCal. This underscores the **importance of a well-trained UDA model in ensuring the effectiveness of PseudoCal**.

Reviewer comment (Experiments):
It is unclear what happens if the same technique is applied in a standard network training setup (without domain shift problems) to calibrate the network.

Author response:
**Without domain shifts between the training/validation data and the testing data**, we anticipate that TempScal with a labeled validation set will perform effectively, **eliminating the need to consider any UDA calibration methods**, including ours. We **do not expect PseudoCal to outperform** TempScal under this situation. To address the reviewer’s concern comprehensively, we conduct experiments on the Clipart (C) domain in DomainNet. The dataset is divided into three equal parts: a training set, a validation set, and a testing set. Model training utilizes the training set, and calibration performance is evaluated on the testing set. Four types of calibration results are considered: "No Calib." (model trained on the training set without calibration), "TempScal-val" (model calibrated with labeled validation data), "PseudoCal-test" (model calibrated with unlabeled testing data), and "Oracle-test" (model calibrated with labeled testing data). Our findings indicate that while PseudoCal remains effective in achieving improved calibration performance, it does not exhibit advantages compared to TempScal when using an IID validation set, **aligning with our expectations.**

|Dataset|Clipart|
|:----|:----|
|No Calib.|17.36|
|TempScal-val|**3.79**|
|PseudoCal-test|3.8|
|Oracle-test|3.16|
|Accuracy (%)|56.65|

Reviewer comment (Theory):
The success of a UDA method in different domain shift situations depends on the size of the gap between the domains. It is expected that the UDA calibration would depend on how well the UDA method manages to close the domain gap, but it is not clear how the proposed method addresses this aspect of the UDA calibration problem.

Author response:
One novelty of our PseudoCal approach lies in adopting a **target-domain perspective** to address the calibration problem in UDA, distinguishing it from existing methods that primarily rely on source data to address the covariable shift challenge. It's essential to note that **in the UDA field, various studies, including many in the recently popular source-free UDA settings, also operate without source data.** 
Similar to previous target-specific works, PseudoCal **relies on a well-trained UDA model that effectively captures the target-domain data structure.** The absence of source data is compensated by leveraging the knowledge embedded in the UDA model. In situations where the UDA model struggles to learn the target-domain structure, the cluster assumption may not hold, potentially damaging the sample-level correspondence between mixed samples and real target samples. This aspect is evident in **Table 6, where, in the I$\to$S task, both SHOT and DINE models exhibit low accuracy, resulting in high calibration errors** even after being calibrated with PseudoCal. This underscores the **importance of a well-trained UDA model in ensuring the effectiveness of PseudoCal**.

Reviewer comment (Evaluation):
Given the first point mentioned in "Strengths", the paper only claimed it as an observation but without further explaination.

Author response:
**A1**: The observation stems directly from the inherent characteristics of the factorized NLL objective. As clarified in **Section 3.1, the optimization process for correct and wrong predictions in the factorized NLL objective inherently seeks contrasting temperature values.** Specifically, **correct predictions aim for a small temperature to sharpen** the softmax probability distribution, while **wrong predictions aim for a large temperature to flatten** the softmax probability distribution. Consequently, when dividing the confidence ranges of (0.0, 1.0) into multiple small bins and ensuring two datasets share the **same correct-wrong statistics (i.e., an identical count of correct or wrong predictions for each confidence bin)**, a highly consistent NLL optimization process is expected. This consistency results in obtaining similar temperatures after the optimization process. We hope this clarification provides a better understanding of the rationale behind this observation.

Reviewer comment (Theory):
The method of using Mixup for pseudo-label generation is not convincing to me, even given the "Analysis" section. (a) While the ground-truth labels are not available, can I just assign random labels for each target sample and then adopt PseudoCal (i.e. How is the UDA inference process helpful)? (b) Since $\lambda$ is a fixed scalar and we do not know whether the target sample prediction result is true or not, what is the point of the last 8 lines in "Analysis" part. (c) The pseudo code in Appendix A does not do what equation (3) says for $y_{pt}$. Also, since $\lambda$ is a fixed scalar in (0.5, 1.0), what does the "if else" statement for pseudo_target_y do as "else" is never visited?

Author response:
**A2**: PseudoCal is a novel and general framework designed to leverage a synthesized labeled pseudo-target set for optimizing TempScal and obtaining an accurate estimation of the Oracle target-domain temperature. The crucial factor ensuring the success of PseudoCal lies in having this pseudo-target set exhibit similar correct-wrong statistics to the real target samples. **However, we cannot know the actual correct-wrong statistics of the real target samples due to the unavailability of target-domain ground-truth labels.** 

Returning to the initial question, while it’s feasible to use randomly assigned labels for calibration with target samples, **there is no assurance of favorable calibration performance because the resulting correct-wrong statistics would be entirely random**. This is demonstrated in **Table 9 under "Pseudo-Label,"** where we employ **hard one-hot target model predictions as labels** for real target samples and apply TempScal, resulting in **substantial calibration errors**.

The effectiveness of PseudoCal with inference-stage mixup relies on establishing a sample-level correspondence between the pseudo-target set and the real target samples, partially guaranteed by the cluster assumption. **The UDA inference process is to obtain predictions** for mixed samples, and **correctness is evaluated** based on these predictions and the corresponding mixed labels. This inference process is critical for **leveraging the cluster assumption** and building the required sample-level correspondence. We have provided comprehensive evidence to illustrate this, including detailed illustrations in **Figure 4 in the Appendix** and extensive quantitative statistics in **Table 10 in the Appendix**. 

**A3**: The "Analysis" part in Section 3.2 aims to provide **an intuitive analysis and explanation of how mixed samples can exhibit similar correct-wrong statistics as real target data**. This is crucial for ensuring that applying TempScal with these mixed samples can yield a temperature similar to the Oracle target temperature. In the **revised paper**, we have enhanced the presentation for better clarity, incorporating **more illustrations** of correct-wrong statistics, as demonstrated in **Figure 1(b) and Figure 4 in the Appendix**. 

It's essential to clarify that we **analyze the dataset-level similar correct-wrong statistics through the sample-level correspondence of correct or wrong predictions. The definition of correctness differs between mixed samples and real samples.** For mixed samples, correctness is assessed based on the predictions of the mixed samples and their corresponding mixed labels. Conversely, for real target samples, correctness is evaluated using the target predictions and unavailable target ground truths. PseudoCal never accesses target ground truths and relies solely on target images and their model predictions to synthesize a labeled pseudo-target set, comprising mixed samples and mixed labels. **For a quantitative analysis of the sample-level correspondence, we kindly refer the reviewer to Appendix D.**

The use of **target ground truths serves exclusively for analysis and explanation purposes** to study the success of PseudoCal. This involves explaining and showcasing similar correct-wrong statistics between the pseudo-target set and real target samples, as discussed in both the Analysis part in Section 3.2 and Appendix D.

**A4**: We appreciate the reviewer's attention to detail. Regarding the concern about the pseudocode, we want to emphasize that **our pseudocode accurately represents the implementation of PseudoCal**, and there is **no inconsistency between the code and the paper**. While **Equation 3** provides a **general formulation of mixup** without specifying whether $y_t$ represents hard one-hot pseudo-labels or soft predictions, we clarify in **"Impact of mix ratio $\lambda$" in Section 4.3** that we use hard one-hot pseudo-labels for $y_t$, **aligning with our pseudocode**.
Additionally, we would like to clarify that we limit the discussion of $\lambda$ to the range of (0.5, 1.0) for simplicity and clarity in our submission because the ranges of (0.0, 0.5) and (0.5, 1.0) are **symmetric** situations. Regarding the **"if else"** statement in the pseudocode, it is important to note that the "else" branch is not typically executed due to the restricted range of $\lambda$ (0.5 to 1.0). Nevertheless, this structure is included **for comprehensive logical coverage**, aiming to **avoid potential bugs** that may arise if the user is not aware of the predefined range for $\lambda$.

Reviewer comment (Reproducibility):
What is the validation set adopted for Temperature scaling in the experiments?

Author response:
**A5**: 
- **For the baseline “TempScal-src,”** representing source-domain calibration, we use **a labeled source validation set** to calibrate the UDA model.
- **For the baseline “Oracle,”** we apply TempScal using model **predictions of unlabeled target data** and the corresponding **target ground truths** for calibration. This serves as an upper bound for target-domain calibration performance as a reference point.
- **For our PseudoCal**, we synthesize a labeled pseudo-target set comprising mixed samples and mixed labels. Subsequently, we **use the UDA model to infer predictions for these mixed samples** and apply TempScal using these predictions and **mixed labels** for calibration.

Reviewer comment (Evaluation):
Table 2~5 shows a phenomenon: Temperature scaling sometimes achieves the best performance. What might be the reason?

Author response:
**A6**: The baseline **“TempScal”** in our initial submission **has been replaced by “TempScal-src,”** explicitly indicating that it is **a source-domain calibration baseline using a labeled source validation set**. The occasional occurrence of “TempScal-src” achieving the best performance is **attributed to the labeled source validation set coincidentally having similar correct-wrong statistics to the unlabeled testing target data.** We want to emphasize that this infrequent phenomenon **does not impact our contributions**. In the **majority of tasks**, as demonstrated in our experiments, **“TempScal-src” shows only marginal calibration improvement** compared to other UDA calibration methods, including our PseudoCal. However, practitioners and researchers in the domain adaptation community are in pursuit of **reliable and consistently effective calibration methods across various UDA tasks and settings**. Evidently, **"TempScal-src" does not align** with this criterion.

Reviewer comment (Presentation):
The style of figure 2 is new to me. However, not good-looking nor clear.

Author response:
**A7**: We apologize if Figure 2 appears unclear or less aesthetically pleasing. We would appreciate it if the reviewer could provide specific feedback on areas that require improvement, allowing us to enhance the figure for better clarity and visual appeal.

Reviewer comment (Novelty):
The novelty is weak because mix-up is a well-known strategy in image classification tasks. From my perspective, the methodology is the same as the previous ICLR 2018 work, "mixup: BEYOND EMPIRICAL RISK MINIMIZATION".

Author response:
We appreciate the reviewer's comments and **respectfully disagree** with the assertion that **“the methodology is the same as the previous ICLR 2018 work [1]”.** 

1) While acknowledging the foundational work of [1] and its impact on the field, we **highlight distinct differences** in how mixup is employed in our work, as **discussed in Section 4.3 "Comparison with training-stage mixup"** and summarized below:

- **Different application stage**: Unlike existing mixup works [1, 2], which leverage mixup during the training stage for model training, **our PseudoCal** approach employs mixup during the **inference stage**, specifically for UDA model calibration, **without altering model parameters**.

- **Different mix ratio ($\lambda$)**: In prior works **[1, 2], effective** mixup commonly utilizes **$\lambda$ values near 1**, typically **randomly sampled** from a beta-distribution, i.e., $\lambda \sim Beta(\alpha, \alpha)$, where $\alpha \in [0.1, 0.4]$. Notably, $\lambda$ values significantly **deviating from 1.0** are considered **ineffective due to the manifold intrusion problem [2, 3]**. On the contrary, **PseudoCal** exhibits **optimal** performance with **fixed $\lambda$** values significantly **deviating from 1.0**. However, employing **$\lambda=1.0$** in PseudoCal (i.e., PseudoCal($\lambda=1.0$)) leads to **significant calibration errors**.

- **Different calibration performance**: We present compelling evidence from two perspectives. **First**, we examine **DINE [4]**, a source-free domain adaptation method **employing the training-stage mixup** strategy [1, 2] during model training. As outlined in **Table 6**, our observations indicate that UDA models trained with DINE still manifest substantial calibration errors. In contrast, our **PseudoCal effectively calibrates the DINE model**, leading to notable reductions in calibration errors. Specifically, the ECE (%) error drops from 21.81 to 12.82 on DomainNet and from 58.85 to 47.76 on ImageNet. Furthermore, when we **substitute our mixup strategy with the training-stage mixup [1, 2] in our PseudoCal framework**, denoting this variant as **"Mixup-Beta" in Table 9**, we observe the superior performance of our mixup strategy. These significant performance differences underscore the distinct nature and advantage of our mixup strategy.

2) Moreover, it is crucial to highlight that our usage of mixup serves as only one technical facet of our framework; **the novelty of our paper extends far beyond this component**, as summarized below.

- **Novel target-domain perspective**: We approach the challenging calibration problem in UDA as **an unsupervised calibration issue in the target domain**, **diverging from the previous** treatment as a **cross-domain covariate shift** problem seen in competing approaches listed in Table 1 of our submission.

- **Novel TempScal objective factorization**: We introduce a **unique factorization of the negative log-likelihood (NLL)** objective used in TempScal, a distinctive feature not identified in relevant UDA model calibration studies compared in Table 1.

- **Novel PseudoCal framework**: Our innovative post-hoc framework, PseudoCal, addresses model calibration in domain adaptation through a novel strategy for synthesizing pseudo-target samples. This sets it apart from competing methods, as thoroughly demonstrated in **Table 1 of our submission**.

- **Comprehensive empirical study**: Our extensive experiments with PseudoCal encompass 10 domain adaptation methods across 5 UDA scenarios. **Notably, we include the calibration under source-free UDA settings and domain-adaptive semantic segmentation for the first time**. PseudoCal consistently exhibits significant outperformance over all competing methods.

Reviewer comment (Evaluation):
The mix-up is conducted at input level, which would introduce artifacts. It would be good to include some visualization in the appendix together with quantitative results.

Author response:
Thank you for your valuable suggestion. We have **added visualizations** of input-level mixup for all UDA benchmarks in **Figure 5 of the Appendix** for a comprehensive understanding.

Reviewer comment (Reproducibility):
How did the author get the results of other methods, cite from the original papers, or re-implement those approaches?

Author response:
Indeed, we meticulously **re-implemented all calibration baselines and UDA methods** featured in our experiments. The UDA methods were trained using their officially released code. As for **calibration methods, we employed the codebase of TransCal [5], accessible at https://github.com/thuml/TransCal**. For each task, we applied all calibration baselines to calibrate the same UDA model. Our experiments underwent five repetitions, and the averaged results are reported in the tables. Our work stands out for providing comprehensive empirical comparisons among all calibration baselines in UDA settings. **Notably, we ventured into evaluating domain adaptive semantic segmentation tasks and source-free UDA settings for the first time.** In these new settings, some existing baselines could not be applied due to their focus on classification tasks and the utilization of source data. Therefore, our comparisons are limited to applicable calibration methods such as Ensemble.

Reviewer comment (Evaluation):
The analysis of the mixup in the inference strategy is not sufficient. The comparison between the training stage mixup strategy in other methods and the inference stage is not obvious, the authors should strengthen your highlight.

Author response:
In Section 4.3, we delve into a detailed analysis of the mixup operation's impact on the calibration performance of our PseudoCal. **Two key implementation-level factors are considered**: the use of hard one-hot labels or soft predictions for the mixup process in Equation 3, and the value of the mix ratio $\lambda$. Empirical results across various UDA methods and settings are presented in **Figure 3(c)-(d)**. Specifically, we highlight the crucial role of $\lambda$ values, **emphasizing in the main text that "a $\lambda$ value closer to 0.5 generates more ambiguous samples, resulting in increased wrong predictions, whereas a $\lambda$ value closer to 1.0 has the opposite effect." For readers interested in detailed quantitative analysis, we provide a comprehensive discussion in Appendix D.**

In comparing our approach to mainstream training-stage mixup [1,2], our goal is to **highlight the novel aspects of our mixup strategy**. We emphasize three key novelties. **Firstly**, we employ **mixup during the inference stage** rather than the traditional training stage. **Secondly**, our mixup strategy exhibits effectiveness with **a fixed $\lambda$ value significantly deviating from 1.0**, a departure from the prevalent use of values close to 1.0 in existing mixup approaches. **Thirdly**, applying training-stage mixup directly to train a UDA model, as seen in UDA methods like **DINE, still results in substantial calibration errors.** Conversely, our PseudoCal can significantly reduce ECE errors for such models, as evidenced by the DINE results presented in **Table 6**. Additionally, in the ablation study of PseudoCal, where we **replace our $\lambda$ usage** with values generated through the beta-distribution employed in existing mixup, the results under **"Mixup-Beta" in Table 9** show that our mixup strategy outperforms traditional mixup.

Reviewer comment (Evaluation):
Writing can be improved. In addition, the experiment results could present more in the “Accuracy” metric, which is also intuitive.

Author response:
Thank you for your valuable feedback on writing. We have revised the manuscript to enhance clarity and provide more detailed results. Regarding the "Accuracy" metric, we have clarified in the **first paragraph of Section 4.2** that, in all tables, "‘Accuracy’ (%) denotes the target accuracy of the **fixed UDA model**." This addition only serves to offer readers a clear understanding of the UDA model's discriminative ability in the target domain. **As indicated in Table 1 and Figure 2, our PseudoCal** method is a **post-hoc** calibration approach that solely **utilizes the UDA model for inference without changing "Accuracy" of the model**. It's worth noting that many existing calibration methods, such as **TempScal, CPCS, and TransCal**, are **also post-hoc** in nature.

Reviewer comment (Reproducibility):
How many pseudo-target samples will be generated? Does the number of n_{pt} be the same with target sample number?

Author response:
Yes, for all experiments, the number of $n_{pt}$ is almost the same as the target sample number $n_t$. Specifically, in the **Implementation details**, we mentioned that “We use the UDA model for **one-epoch inference with mixup to generate the labeled pseudo-target set**.” Additinoally, as indicted in our pseudocode in **Appendix A**, during the inference process, we **only mix real target samples that have different pseudo labels**, i.e., cross-cluster perturbation. Therefore, in general, **$n_{pt}$ would be slightly smaller than $n_t$** due to **ignoring the pairs of the shuffled real target samples that have the same pseudo label**.

Reviewer comment (Experiments):
To find the value of mix ratio \lambda, the authors analyze the ECE on different UDA scenarios. However, it seems that the black-box source-free scenario does not have a corresponding analysis.

Author response:
We appreciate the thorough examination of our ablation experiments in Table 9 and analysis experiments in Table 10 (Appendix). Both tables maintain a consistent setting for UDA methods. In **our initial submission**, we presented the black-box source-free method **DINE** and the white-box source-free method **SHOT under the same parent UDA setting, i.e., source-free UDA**. Consequently, we only reported SHOT ablation results. **Following the valuable suggestion, we have included the results of DINE in both Table 9 and Table 10** to facilitate comprehensive comparisons **without altering our initial conclusions**.