Paper: IW-GAE: Importance weighted group accuracy estimation for improved calibration and model selection in unsupervised domain adaptation (ICLR-2024)

Reviewer comment (Theory):
This paper proposes to achieve calibrated accuracy from the perspective of group accuracy. However, it seems to me that intuitively, group accuracy is more coarse-grained than the accuracy of each individual. Thus, I am curious that, will this direction of group accuracy lead to a lower upper limit than the other branches of method?

Author response:
We sincerely appreciate the Reviewer WXHo for the insightful question that deals with the main motivation of our method. As the Reviewer WXHo commented, estimating group accuracy rather than individual accuracy might seem more coarse at the first glance. However, in Section 4.1 of the revised manuscript which is a summary of the result that was originally located in Appendix B.1, we showed that estimating group accuracy is more statistically favorable than estimating individual accuracy in most machine learning scenarios where only a single label is available for each input value. Specifically, the MLE of group accuracy has a lower expected mean-squared error than the MLE of individual accuracy under relatively loose conditions. 

Due to its significant importance for motivating our method, the revised manuscript includes the newly added Section 4.1 which summarizes the motivation for estimation group accuracy.

Reviewer comment (Theory):
If I am not wrong, the logic of this paper's proof seems to be, (1) the proposed optimization algorithm can optimize over the upper bound of source group accuracy estimation error, and (2) the source group accuracy estimation error serves as an upper bound of the target group accuracy estimation error. Thus, because of (1) and (2) one by one, the proposed optimization algorithm works. My question with respect to this is that, are your two upper bounds tight enough to make the approach theoretically meaningful? Taking Eq. 5 as an example, while I do not check the math very carefully, it seems that the target group accuracy estimation error is upper bounded by the source group accuracy estimation error multiplied with both M and the other item in the bracket. Thus, both these two items, if large to a certain scale, will make the optimization over the upper bound sub-meaningful. I understand that the authors have discussed that they will tighten the bound by "bound the maximum and minimum values of IW". However, I think this part should be elaborated as the tightness of the bound is important from my perspective. Besides, I also appreciate if the other impacts of this "bound the maximum and minimum values of IW" operation can be discussed. For example, will this operation hurt the current theoretical flow?

Author response:
We thank the Reviewer WXHo for the insightful question associated with a fundamental theoretical aspect of the proposed method. Yes, as the Reviewer WXHo commented, tightness of both bounds is crucial for connecting the optimization error and estimation errors of the target and source group accuracy. Due to its central importance, we have added discussions about the conditions when the bounds are tight in the corresponding locations (page 5 for the relationship (1) and page 7 for the relationship (2)). 

We also have summarized the results of empirical analyses of these bounds to support our main theoretical results for connecting the optimization error in IW-GAE and the group accuracy estimation error in the target domain. Please see the “common response for empirical evidence supporting theoretical claims” for more details about the empirical evaluation. 


Again, we appreciate a very great question from the Review WXHo since bounding values usually introduces a gap between theory and practice. First, we apologize for our original statement "bound the maximum and minimum values of IW" which induces misunderstanding. It should be changed to "bound the maximum and minimum values of an IW estimation." Given the corrected statement, bounding IW estimation value influences only the estimator, so it does not affect any theoretical guarantee using properties of IWs in the paper (e.g. Proposition 4.2 and inequalities (5) and (14) in the revised manuscript). However, bounding an IW estimation value can reduce its search space from its confidence interval to a bounded CI, which may increase the optimization error. However, in practice, we observed that bounding helps to improve the performance as we intended. 

Due to its importance, we have added this discussion in the last paragraph of Section 4.2 of the revised manuscript.

“””
We note that bounding an IW estimation value only affects the estimator, which does not affect any theoretical guarantee based on properties of true IWs such as Proposition 4.2 and inequalities in (5) and (14). However, we remark that bounding an IW estimation may increase the optimization error due to reduced search space, although we observe that it works effectively in practice. 
“””

Reviewer comment (Presentation):
The legend and meaning in Figure 1 are unclear and difficult to comprehend. The annotation for Figure 1 is excessively lengthy and could benefit from moving some sections to the main body of the paper as illustrative examples.

Author response:
(Comment) “The legend and meaning in Figure 1 are unclear and difficult to comprehend. The annotation for Figure 1 is excessively lengthy and could benefit from moving some sections to the main body of the paper as illustrative examples.”

(Response) Thanks for the great suggestion. As per Reviewer 4Yn5’s comment, we have significantly improved the caption of Figure 1 by delivering the core ideas of encouraging consistency between two different estimators and an ideal case for group accuracy estimation without unnecessary details. We also have moved some descriptions to the main body of the revised manuscript for better illustration (kindly see page 2 of the revised manuscript). Specifically, we have changed the caption of Figure 1 as follows.

“””
In Figure 1(a), a shaded area for the IW-based estimator represents possible estimations from IWs in the confidence interval.
Figure 1(b) illustrates both ideal and failure cases of IW-GAE with nine data points (red diamonds) from three groups (gray boxes). Group 1 is desirable for model calibration where the group accuracy estimation (a blue rectangle) well represents the individual expected accuracies of samples in the group. Conversely, group accuracy estimation could inaccurately represent the individual accuracies in the group due to a high variance of accuracies within the group (group 2) and a high bias of the estimator (group 3).
“””

Reviewer comment (Experiments):
The experiments conducted in the main paper don't sufficiently verify the proposed method's effectiveness. Other datasets, such as DomainNet, and other baselines in Unsupervised Domain Adaptation (UDA), should be incorporated.

Author response:
(Comment) “The experiments conducted in the main paper don't sufficiently verify the proposed method's effectiveness. Other datasets, such as DomainNet, and other baselines in Unsupervised Domain Adaptation (UDA), should be incorporated.”

(Response) We have performed two additional experiments with another large-scale dataset (DomainNet) and an additional UDA base model (maximum classifier discrepancy). IW-GAE consistently achieves the best performance as in other benchmark experiments. For more details, kindly see the “common response for the extensiveness of experiments.”

Reviewer comment (Presentation):
Several analyses and discussions residing in the appendix are crucial to demonstrating your method's effectiveness. It is advised to relocate these sections to the main body of the paper.

Author response:
(Comment) “Several analyses and discussions residing in the appendix are crucial to demonstrating your method's effectiveness. It is advised to relocate these sections to the main body of the paper.”

(Response) We sincerely appreciate the Reviewer 4Yn5 for the valuable suggestion that helps us to significantly improve the writing of the manuscript. Following the suggestion, we have summarized the motivation for estimating group accuracy that was originally discussed in Appendix B.1 to the main body. In addition, we also summarized analyses of correlations between the optimization error, the source group accuracy estimation error, and the target group accuracy estimation error discussed in Appendix E.7 of the original manuscript in the main body. We believe the insightful feedback about the layout of the manuscript from the Reviewer 4Yn5 significantly improves the revised manuscript. 

For more details about the changes we made, please see the “common response for the motivation for estimating group accuracy” and “common response for empirical evidence supporting theoretical claims.”

Reviewer comment (Experiments):
The paper asserts that the proposed method is not applicable to fixed large-language models in IW-GAE. Thus, a more comprehensive discussion and additional experiments are recommended.

Author response:
(Comment) “The paper asserts that the proposed method is not applicable to fixed large-language models in IW-GAE. Thus, a more comprehensive discussion and additional experiments are recommended.”

(Response) Kindly see the “common response for experimental results with large-language models” which explains not only IW-GAE but also all other IW-based methods (IWCV, DEV, TransCal, CPCS) fail to improve a vanilla method and the implications of the result for future research.

Reviewer comment (Novelty):
The contribution's novelty is limited. The paper employs importance weighting to simultaneously address both the model calibration and selection problems. However, similar works, such as (You et al., 2019) for model selection and (Wang et al., 2020; Park et al., 2020) for model calibration, have been previously published. The primary difference between this paper and previous works appears to be the introduction of bin-wise importance weighting, as proposed in (Park et al., 2022). In summary, this paper mainly replaces the use of importance weighting in prior works with a more recent advanced importance weighting technique.

Author response:
(Comment) “The contribution's novelty is limited. The paper employs importance weighting to simultaneously address both the model calibration and selection problems. However, similar works, such as (You et al., 2019) for model selection and (Wang et al., 2020; Park et al., 2020) for model calibration, have been previously published.”

(Response) We appreciate the Reviewer Yb2X for pointing out the unclear delivery of the novelty of the paper. As a fundamental method in statistics and simulation, importance weighting has been used to derive unbiased estimators under distribution shifts in various contexts of machine learning such as reinforcement learning, causal inference, and domain adaptation. In the context of model calibration and model selection in UDA, importance weighting is used to estimate various quantities of interest in the target domain such as calibration measures (Wang et al., 2020; Park et al., 2020) and model accuracy (Sugiyama et al., 2007; You et al., 2019). However, different from the previous research, we define a new measure of group accuracy and provide a concrete algorithm for accurately estimating the newly defined measure of group accuracy with thorough theoretical support. 

In addition to the novelty of defining a new measure, our approach can solve important tasks of model calibration and model selection at the same time with attractive properties. 
For model calibration, since our group accuracy estimator error is directly connected to the calibration error and its bound is theoretically shown, the proposed method can guarantee a bounded calibration error in the target domain. A bounded calibration error has not been guaranteed in the previous literature, which is a significant theoretical contribution. 
For model selection, we agree that the proposed method may seem similar to the previous work (Sugiyama et al., 2007, You et al., 2019) in the sense that all methods aim to predict the model accuracy in the target domain. However, our approach has an additional regularization factor that aims to accurately estimate the accuracy of each group, which gives more fine-grained estimation ability. The effectiveness of this additional regularization is well-supported by empirical results that IW-GAE achieves state-of-the-art performances in hyperparameter selection and checkpoint selection which are important model selection tasks. 

To emphasize the novelty of IW-GAE compared to existing approaches, we have added the following discussion in Section 4 of the revised manuscript.


“””
Once we obtain an IW estimation and a group assignment with methods described in Section 4.1, IW-GAE can estimate the group accuracy that can be used to simultaneously solve model calibration and model selection tasks with attractive properties. Specifically, for model calibration, previous approaches (Park et al., 2020; Wang et al., 2021) depend on a temperature scaling method (Guo et al., 2017) that does not provide theoretical guarantees about the calibration error. In contrast, IW-GAE uses the group accuracy estimation as an estimate of confidence. Therefore, due to the guarantees about the group accuracy estimation error (cf. Proposition 4.2 and (5)), IW-GAE enjoys a bounded calibration error. For model selection, IW-GAE uses average group accuracy computed with the dataset in the target domain as a model selection criterion. While the previous approaches (Sugiyama et al., 2007; You et al., 2020) also aim to estimate the model accuracy in the target domain, IW-GAE considers an additional regularization encouraging accurate group accuracy estimation for each group.
“””

Reviewer comment (Experiments):
The empirical evaluation is inadequate, rendering the conclusion less reliable. The paper only examines one UDA method, MDD, on a single UDA dataset, Office-Home, for assessing the proposed method, IW-GAE. This limited scope does not provide sufficient empirical evidence for the effectiveness of IW-GAE. It is recommended to expand the experiments to encompass more UDA methods and datasets, following relevant works (Wang et al., 2020; You et al., 2019). In conclusion, the paper lacks the necessary empirical support for IW-GAE.

Author response:
(Comment) “The empirical evaluation is inadequate, rendering the conclusion less reliable. The paper only examines one UDA method, MDD, on a single UDA dataset, Office-Home, for assessing the proposed method, IW-GAE. This limited scope does not provide sufficient empirical evidence for the effectiveness of IW-GAE. It is recommended to expand the experiments to encompass more UDA methods and datasets, following relevant works (Wang et al., 2020; You et al., 2019). In conclusion, the paper lacks the necessary empirical support for IW-GAE.”

(Response) We sympathize now for confusing Reviewer Yb2X about the comprehensiveness of experiments in the manuscript. In the Appendix of the original manuscript, we included additional experiments with a different UDA baseline (CDAN) and a different dataset (VisDa 2017). Also, we performed another important model selection task of choosing the best checkpoint during training. In these additional experiments, IW-GAE consistently achieves the best performances among important baselines we consider in the paper (Guo et al., 2017; Park et al., 2020; Wang et al., 2020; Sugiyama et al., 2007; You et al., 2019). 

In the revised manuscript, as per Reviewer Yb2X’s valuable suggestion, we perform two additional benchmark experiments with a different large-scale dataset (DomainNet) and a different base UDA method (maximum classifier discrepancy) in order to match the comprehensiveness of our empirical verification with previous works (You et al., 2019; Wang et al., 2020; Park et al., 2020). In the new benchmarks, IW-GAE consistently achieves the best performances. For more details, kindly see the common response “common response for the extensiveness of experiments.”

Reviewer comment (Experiments):
The proposed method may not be effective and practical, significantly diminishing the paper's contribution. While the empirical evaluation in the main text demonstrates the effectiveness of IW-GAE with MDD, the evaluation in the appendix reveals that IW-GAE does not outperform TransCal (Wang et al., 2020) with CDAN. Moreover, IW-GAE is considerably more complex, as it involves multiple hyperparameters compared to the hyperparameter-free TransCal (Wang et al., 2020) and DEV (You et al., 2019). IW-GAE requires at least three sensitive hyperparameters to be configured: temperature, the number of accuracy groups, and the number of bins, which does not guarantee its suitability for the model selection problem. In conclusion, the current, albeit insufficient experiments suggest that IW-GAE is not competitive and practical when compared with existing solutions.

Author response:
(Comment) “the evaluation in the appendix reveals that IW-GAE does not outperform TransCal (Wang et al., 2020) with CDAN.”

(Response) We kindly refer Reviewer Yb2X to Appendix E.2 for additional experiments with a different base model (CDAN) where IW-GAE outperforms TrasCal (Wang et al., 2020) by 2%, which was given in the original manuscript. In addition, in our new experiments that we performed thanks to the suggestion by Reviewer Yb2X, IW-GAE consistently outperforms TransCal in different benchmarks with a different dataset (DomainNet) and a different base model (MCD).

(Comment) “IW-GAE is considerably more complex, as it involves multiple hyperparameters compared to the hyperparameter-free TransCal (Wang et al., 2020) and DEV (You et al., 2019). IW-GAE requires at least three sensitive hyperparameters to be configured: temperature, the number of accuracy groups, and the number of bins, which does not guarantee its suitability for the model selection problem. ”

(Response) We now agree with Reviewer Yb2X’s confusion about hyperparameters included in IW-GAE and not effectively referring to relevant analyses that we performed in the original submission. First of all, we want to note that hyperparameters involved in IW-GAE are only the number of groups and the number of bins, not the temperature parameter. Specifically, the temperature parameter is optimized within the search space in IW-GAE, which is the same as with the previous methods (Wang et al., 2020, You et al., 2019). Specifically, in the literature (Wang et al., 2020, You et al., 2019), the search space is defined as a box constraint, i.e., setting the maximum and minimum values of the temperature parameter. However, in IW-GAE, it is defined as a discrete space due to the non-smoothness of the optimization problem. From this perspective, we agree that IW-GAE involves a more complex optimization procedure compared to previous literature. However, we believe that achieving state-of-the-art performances in both model calibration and selection tasks under three popular UDA methods (MDD, CDAN, and MCD) on three different datasets (OfficeHome, VisDa-2017, and DomainNet) can account for the more complex optimization procedure involved in IW-GAE.

Also, we performed the sensitivity analysis with respect to all hyperparameters involved in IW-GAE (the number of groups and the number of bins) in Appendix E.6, which shows the robustness of performances of IW-GAE under changes in these hyperparameters. Also, we want to emphasize that we perform all experiments with three different architectures on three different datasets with a single fixed value of the hyperparameter, which again shows the robustness of IW-GAE for the hyperparameter choice. As it might not be easy to detect the citation of additional experiments that include sensitivity analysis in the original manuscript, we have added a new section titled “5.3. Qualitative evaluation, ablation study, and sensitivity analysis.”

Reviewer comment (Presentation):
The paper's presentation lacks clarity due to weak organization and missing information, particularly when compared to the well-structured writing in two highly relevant importance-weighting works (Wang et al., 2020; You et al., 2019) in the context of UDA. The current submission includes excessive background and proofs while omitting essential details about the algorithm and empirical evaluations. Furthermore, the motivation and novelty of this paper in comparison to existing importance-weighting works on model calibration and model selection remain unclear.

Author response:
(Comment) “The paper's presentation lacks clarity due to weak organization and missing information, particularly when compared to the well-structured writing in two highly relevant importance-weighting works (Wang et al., 2020; You et al., 2019) in the context of UDA. The current submission includes excessive background and proofs while omitting essential details about the algorithm and empirical evaluations. Furthermore, the motivation and novelty of this paper in comparison to existing importance-weighting works on model calibration and model selection remain unclear.”

(Response) We appreciate the Reviewer Yb2X for pointing out the weak organization that includes the important details in the Appendix and does not include the essential details about the algorithm and evaluations. Following the Reviewer Yb2X’s valuable feedback, we have added the pseudocode of the proposed method to provide full details of the algorithm required to implement the proposed method in Appendix C.3 based on clearly written pseudocode of the previous work (Wang et al., 2020). In addition, we also have added pseudocodes for evaluating methods for model calibration and model selection in Appendix C.3. Please, kindly see Algorithms 1, 2, and 3 on pages 17-18 of the revised manuscript. We believe these additions will greatly help readers to understand the proposed method in detail. 

Also, thanks to valuable feedback from the Reviewer Yb2X, we have significantly simplified the explanation of the previous method (Park et al., 2022) by excluding unnecessary details and derivations. For the change, kindly check pages 3-4 of the revised manuscript. 

Finally, kindly see the first response for the novelty of the paper compared to relevant works and the “common response for the motivation for estimating group accuracy”  for the motivation for estimating group accuracy.