From: NeurIPS.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept

From: NeurIPS.cc/2022/Conference/Paper6146/Area_Chair_z3wR
metareview: The paper proposes a new method for leveraging a zoo of pre-trained models for improving OOD generalization in an efficient way. The reviewers agree that the results are strong.
recommendation: Accept
confidence: Certain
award: No

From: NeurIPS.cc/2022/Conference/Paper6146/Authors
title: Further Discussion with Reviewers
comment: Dear Reviewers,

We hope that our responses have addressed the concerns raised in your review. If there remain concerns or if you have more questions, we are more than happy to provide additional clarification. Thank you so much for your time!

Sincerely,

Authors of Paper 6146

From: NeurIPS.cc/2022/Conference/Paper6146/Authors
title: General Response
comment: We thank reviewers for their time and insightful positive feedback. 

We are encouraged that reviewers found our paper technically solid and on an interesting and relevant topic (**R1**); the method being technically solid and reasonable (**R3**). We are happy that they regarded our paper as well-written and easy to follow (**R2**), and having an excellent presentation (**R3**). We are pleased that they found it contains rigorous comparisons and thorough methodological ablations and shows strong empirical gains (**R1**); extensive experiments outperforming the previous SOTA with high efficiency (**R2**); experiments extensive, comprehensive, and convincing, and method outperforming SOTA baselines on most datasets (**R3**).

We have conducted a few more experiments to answer reviewer questions. We are happy to answer any further queries.

**Reviewer Abbreviations**: We have used the order as the reviewer's identity: z3dR is R1, SbCP is R2, and gvF7 is R3.

From: NeurIPS.cc/2022/Conference/Paper6146/Authors
title: Reply to R3 (Part 2)
comment: 
___
**Q12**: Why IID ranking baselines perform so bad on TerraIncognita? Is fine-tuning top linear classifiers a good metric {our interpretation: ground truth} on this dataset?

**A12**: 

> Why IID ranking baselines perform so bad on TerraIncognita?

TerraIncognita dataset consists of images captured by wild cams in several different environments. The task in this dataset is to learn from a few environments and generalize to novel ones. Apart from the difference in environments, images are taken using different techniques (e.g., infrared vs. flashlight for night photography) and have many challenges (e.g., poor illumination, motion blur, the small size of the region of interest, occlusion, camouflage) [2]. Similarly, most images have small foreground to background pixels ratio. These unique challenges make this dataset especially hard for models trained on relatively curated datasets. 

The unique nature of this dataset and abundance of background pixels make it particularly hard for IID ranking metrics. IID metrics only measure inter-class discriminability, not inter-domain stability. This makes IID metrics more brittle for TerraIncognita.

> Is fine-tuning top linear classifiers a good metric on this {TerraIncognita} dataset?

To answer this question, we performed an ablation study that compared model ranking metrics with fully fine-tuning results. To get the ground truth, we fully fine-tuned our model zoo on TerraIncognita adopting a leave-one-domain-out cross-validation setup in [1]. Then, we computed the $\tau_w$ of different IID ranking metrics and ZooD and compared it with $\tau_w$ using the results of the fine-tuning top classifier as the ground truth. The comparison is shown in the following table.  

If linear-classifier-based ground truth is a good metric, we should expect the score to remain the same or get higher. We observe this trend for all methods except for LogME (0.02 to -0.05). Our approach also has a significant improvement (0.46 to 0.69). These results illustrate that linear-classifier-based evaluation is at capturing the performance correlation.

| Method      | $\tau_w$ for Fine-tuning Classifier           | $\tau_w$ for Fine-tuning Full Model   |
| ----------- | ----------- | ---------|
| H-Score     | -0.20           | -0.14     |
| LEEP        | 0.02       |0.08     |
| NCE         | -0.44        |-0.35     |
| LogME       | 0.02        |-0.05     |
| ZooD        | 0.46        |**0.69**     |


___
**Q13**: ZooD uses a large set of PTMs including the newly proposed Transformer-based models, while the baselines (e.g., Gulrajani and Lopez-Paz [18]) use ResNet-50. My main concern is that the comparisons may be unfair due to differences in the performance of different architectures. So, do the different network architectures affect the results?

**A13**:
OoD generalization is a well-explored area of research and many algorithms have been proposed that aim to learn generalizable features from multiple distributions. All of these algorithms focus on the algorithmic side of OoD. However, it is not clear whether they bring significant improvements compared with simple empirical risk minimization [1]. On the other hand, recent works have shown strong OoD performance of PTMs, albeit in perturbations and noise settings only (e.g. Radford et. al. [3]). However, the selection of a good-performing OoD model in a large-scale zoo of available models is prohibitively expensive. 

To tackle this problem, we proposed a computationally cheap model ranking method. For instance, on PACS dataset, our ranking method can select the best model in 0.27 hours compared with 2662.27 hours required for brute-force fine-tuning. Our method can be regarded as a complement to other OoD algorithms. After selecting the top-ranked models, different OoD algorithms can be adopted to fine-tune the models.

To further show the effectiveness of our method, we performed an ablation study with ResNet-50 models only. However, these models are trained differently (e.g., adversarially trained models). For this experiment, we controlled the architecture (e.g., ResNet-50 only) and OoD training algorithm (e.g., ERM only). We compared the best-selected model with a simple standard baseline (PyTorch ResNet-50 used by most OoD papers) in the following table. 

| Dataset       | PACS | VLCS | Office-Home | TerraInc. | DomainNet |
| ------------- | ---- | ---- | ----------- | --------- | --------- |
| Baseline      | 66.9 | 76.7 | 67.7        | 31.0      | 31.1      |
| Ours (Single) | 91.7 | 79.1 | 76.1        | 32.5      | 46.3      |


## References
[1] Gulrajani, Ishaan, and David Lopez-Paz. In Search of Lost Domain Generalization. In ICLR, 2021.

[2] Beery, Sara, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In ECCV, 2018.

[3] Radford, Alec, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.

From: NeurIPS.cc/2022/Conference/Paper6146/Authors
title: Reply to R3 (Part 1)
comment: Thank you for your time and detailed and insightful feedback. We are happy that you find our method technically solid and reasonable, experiments comprehensive and convincing, and outperforming the previous SOTA on most datasets. We answer your questions here. 

___

**Q9**: The complexity of computing proposed metric on m domains is O(m^2), which will be inefficient when there are a large number of domains. Although it is still much faster than brute-force finetuning, how does it compare with other non-finetuning methods?

**A9**: 

>The complexity of computing the proposed metric on m domains is O(m^2) which will be inefficient when there are a large number of domains.

In real-world applications, given an OoD task with $m$ domains, we only need to evaluate each PTM m times, each time treating one held-out domain as validation domain while aggregating all remaining $(m - 1)$ domains as the training data. Hence, the complexity of computing the proposed metric on $m$ domains is $O(n \cdot m)$, where $n$ is the number of models in the zoo.

> Although it is still much faster than brute-force finetuning, how does it compare with other non-finetuning methods?

Selecting the best PTM for a downstream task is an important problem. Recently, several ranking metrics have been devised. However, all of these methods assume IID settings. In this work, we propose a ranking metric that works under OoD settings. To show that our metric works better under OoD conditions, we performed a comprehensive set of experiments and showed that our method not only outperforms IID ranking metrics but it is also more stable and consistent.  

More specifically, IID ranking methods might work well for some OoD datasets but they perform worse for other OoD datasets. For instance, LogME works well for VLCS ($\tau_w = 0.79$) but it completely fails on TerraIncognita ($\tau_w=0.02$). Our method is consistent and it works well across the datasets (e.g., $\tau_w = 0.80$ for VLSCS as well as $\tau_w=0.46$ for TerraIncognita). The consistency of our metric makes it useful for OoD settings. This is because we do not have access to ground truth in the real world. 

Since LogME and other non-finetuning-based metrics are not designed for OoD settings (and they do not perform consistently under OoD settings), a direct comparison of our method with LogME may not be desirable. That's why we compared our method with fine-tuning, i.e., actually fine-tuning all the models with leave-one-domain-out cross-validation on the datasets and then ranking the models. 

___
**Q10**: Authors better put the results of finetuning the whole network in table 2 to improve readability.

**A10**: Thank you for the recommendation. We have conducted an experiment on fine-tuning the full model (top-1) on benchmark datasets. The results are presented in the following table. In summary, fine-tuning the top classifier with feature selection achieves better performance on benchmark datasets except for TerraIncognita. We discuss the TerraIncognita case in A12. Empirically, we find if a PTM is suitable for a given OoD task, fine-tuning the top classifier has better OoD generalization than fine-tuning the full model. 

| Dataset                          | PACS        | VLCS     | Office-Home  | TerraInc. | DomainNet|
| -------------------------------- | ----------- | ---------| -------------| --------- | -------|
| Single (fine-tune classifier)    | 96.0        | 79.5     | 84.6         | 37.3      | 48.2   |
| Single (fine-tune full model)    | 90.6        | 79.1     | 83.4         | 52.6      | 49.7|
| F. Selection                     | 96.3        | 80.6     | 85.1         | 42.3      | 50.6|


___

**Q11**: It would be better if authors report the finetuning results of the PTM selected by the IID ranking baselines (H-Score, LogME, LEEP and NCE) to better compare the ranking methods.

**A11**: 
This question is equivalent to checking whether these ranking methods can select the same top-1 model on different Zoos and datasets. For strict verification, we need to traverse (2^35-1) cases (all the possible subsets of PTMs in the zoo). On the other hand, we can compare the correlation between the orders of PTMs from different ranking methods to check whether these ranking methods are similar. The worst case for the ordering consistency ($\tau$) of ZooD and other methods is as follows:

|    |H-Score| LogME | LEEP | NCE |
|----|--------| -----|-----| -----|
|**ZooD**| 0.20 | 0.49 | 0.42|-0.22|

The above results show that ZooD and the IID ranking methods have significantly different ranking results. 

___

From: NeurIPS.cc/2022/Conference/Paper6146/Authors
title: Reply to R2 (Part2)
comment: 
___

**Q7**: Regarding the computational efficiency, the paper compared the proposed ZooD with brute-force finetuning. When comparing with a simple ensemble, such as LogME, does ZooD still have advantages?

**A7**: The selection of a top-performing PTM for a downstream task is a critical problem. Recently, several ranking metrics have been proposed. However, all of these methods assume IID settings. In this work, we propose a ranking metric that aims to measure the OoD generalization of a PTM. To show that our metric works better under OoD conditions, we performed extensive experiments and demonstrated that our method not only outperforms IID ranking metrics but is also more stable and consistent.  

More specifically, IID ranking methods may perform well on some OoD datasets but they are not consistent. For instance, LogME works well for VLCS ($\tau_w = 0.79$) but it completely fails for TerraIncognita ($\tau_w=0.02$). Our method is consistent and works well across the datasets (e.g., $\tau_w = 0.80$ for VLSCS as well as $\tau_w=0.46$ for TerraIncognita). The consistency of our metric makes it useful for OoD settings as we do not have access to OoD samples during the training procedure. 

Furthermore, different from IID methods, our method adopts leave-one-domain-out cross-validation, which is crucial for OoD tasks. By evaluating held-out domains, we can filter out models that fixate on domain-specific features (Please refer to Appendix B.5).

Since LogME and other metrics are not designed for OoD settings (and they do not perform consistently under such settings), a direct comparison of our method with LogME may not be desirable. That's why we compared our approach with leave-one-domain-out cross-validation-based fine-tuning. 

___

**Q8**: In Line 192, do you refer to the wrong equation? In Table 2, Feature selection contributes more to the performance improvement compared to model Ensemble. What if the feature selection approach is directly applied on the single model?

**A8**: Thanks for pointing this out. It should be the equation between Line 191 and 192. Applying feature selection to a single model does not achieve consistent improvement. Our feature selection method is mainly designed for the ensemble of multiple PTMs. The reason is model ensemble will bring in invariant but non-informative features that are irrelevant to the classification problem. However, forcing feature selection in a single model violated this logic and can not guarantee consistent improvement over all datasets.

From: NeurIPS.cc/2022/Conference/Paper6146/Authors
title: Reply to R2 (Part1)
comment: Thank you for your time and insightful comments. We are encouraged by your positive feedback and happy that you find our paper well-written and easy to follow, with extensive experiments beating previous best approaches with high efficiency. We answer your questions here. 


**Q4**: Regarding the experiment results shown in Figure 2, the proposed ZooD approach does not show much better performance when compared to LogME, except on Terralncognita. However, the ZooD requires a linear model with Gaussian error for evaluation.

**A4**: 
For a rigorous evaluation and comparison of our method, we utilized two different comparison measures: Kendall's coefficient and weighted Kendall's coefficient. Kendell's coefficient ($\tau$) measures relative correlation over all of the models in a zoo. Therefore, it is a good measure of the overall correlation between the ranking of selected models and ground truth. 

However, the ranking of top-performing models is more important than the rest. For this reason, we also report weighted Kendell's coefficient ($\tau_w$), which assigns more weights to the right selection of top models.

First, our method has a better $\tau$ compared with LogME on all seven OoD datasets. For instance, on Office-Home, ZooD has Kendall's correlation of 0.85 with ground truth compared with 0.77 for LogME. Similarly, on TerraIncognita and DomainNet, our method has a $\tau$ of 0.40 and 0.73 compared with 0.04 and 0.54 for LogME.

Second, our method is more stable and consistent than other feature-based model selection methods. For instance, H-Score has a $\tau$ score in the range of $[-0.08 \to 0.67]$, compared with our method's range of $[0.40 \to 0.85]$. The consistency of a transferability metric across different datasets is critical since the purpose of a transferability metric is to estimate performance on a new dataset without having access to ground-truth accuracy. Whenever an estimation metric is inherently unstable, it is hard to determine its reliability for a new dataset.

Third, as shown in Table 1 (a, b), our method also outperforms all of the IID ranking methods on a better benchmark of $\tau_w$. As discussed earlier, weighted Kendall's coefficient is a better measure as it shows the effectiveness of a method for the selection of top-performing models. Our method outperforms previous algorithms on most OoD datasets. For instance, our approach has a $\tau_w$ of 0.46 compared to 0.02 for LogME.

Finally, both LogME and ZooD require a linear model to compute the metric. However, LogME does not consider the OoD scenario and does not align with our assumption of distribution shift.

In summary, our method outperforms the SOTA IID ranking method (LogME) for both $\tau$ and $\tau_w$ measures. It is also more stable and consistent compared with other IID ranking methods.

___


**Q5**: For the experiment results in Table 2, the experiment result of the Single model outperforms the previous SOTA by a large margin. However, the Single model here indicates a fine-tuning model that is selected from the PTMs. I am curious why such a baseline is not included in the table. It seems unfair to compare a selected top-1 model from a zoo of 35 models with a single model on the 5 datasets.

**A5**: In this work, we propose a ranking metric that aims to measure the OoD generalization of a PTM. ***The strong absolute performance against prior work shows that our method can exploit the power of a zoo of PTMs*** (Table 2). Using a zoo with good PTMs here does not lead to unfair comparisons. 
(1) Given a zoo of PTMs, our method aims to exploit the power of the zoo for OoD generalization.
(2) Our method can be regarded as a complement to other OoD algorithms. After selecting the top-ranked models, different OoD algorithms can be adapted to fine-tune the models.
(3) We built a model zoo with 35 PTMs. We can further increase the power of the model zoo by adding more PTMs. This does not affect the purpose of our experiments. From Table 2, we can find that: (i) Using a good PTM zoo can surpass single-model SOTA results; (ii) Our method fully exploit the power of a given PTM zoo.

___

**Q6**: Regarding the ensemble results in Table 2, do you have the performance comparisons of the Ensemble approach with the LogME approach?

**A6**: Although the ensemble is one component of ZooD, the ensemble alone does not provide significant improvement. However, an ensemble of selected models combined with feature selection will lead to better OoD generalization. As LogME is a ranking method primarily designed for IID cases, we compare ZooD with LogME on model ranking performance. Since ZooD is the first framework for OoD generalization by leveraging a zoo of PTMs, to our best knowledge, we only compare our model ranking method with those designed for IID tasks, which is shown in Figures 2 and 3.


___

From: NeurIPS.cc/2022/Conference/Paper6146/Authors
title: Reply to R1 (Part 2) 
comment: ___

**Q3**: The benefits of ensemble over just using the best pre-trained model selected seem fairly minimal.

**A3**: 
As discussed in A2, PTMs can help improve OoD generalization. However, the selection of the best PTM is computationally prohibitive. The main contribution of our work is a computationally cheap model ranking method. 

The second question we tackled is how to efficiently utilize these models. Two simple methods are fine-tuning the top-1 model or assembling top-k models. A simple model ensemble, however, provides fairly minimal improvement because it may introduce invariant but noisy features. To solve this problem, we combine the model ensemble with the proposed feature selection algorithm. This approach is the first step towards efficient utilization of model zoo for OoD generalization. 

The effect of model ensemble combined with feature selection is more prominent on more complex and large-scale datasets such as DomainNet. The DomainNet dataset consists of 6 diverse domains (clip, infographics, paint, quick-draw, real, and sketch) with 586,000 images belonging to 345 classes. The best-selected model by our ranking method can achieve 48.2%: a 1.7% absolute improvement over the previous best). On the other hand, our method achieves an average of 50.6%, double the absolute improvement brought by the best-selected model. 

Similarly, our method brings an absolute improvement of 5.1% (37.2% to 42.3%) on TerraIncognita compared with the single best model. 
___


## References 

[1] Radford, Alec, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.

[2] Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Xiaodong Song. Pretrained transformers improve out-of-distribution robustness. In ACL, 2020.

[3] Isabela Albuquerque, Nikhil Naik, Junnan Li, Nitish Shirish Keskar, and Richard Socher. Improving out-of-distribution generalization via multi-task self-supervised pretraining. ArXiv, abs/2003.13525, 2020.

[4] Mingyang Yi, Lu Hou, Jiacheng Sun, Lifeng Shang, Xin Jiang, Qun Liu, and Zhi-Ming Ma. Improved ood generalization via adversarial training and pre-training. In ICML, 2021.

[5] Gulrajani, Ishaan, and David Lopez-Paz. In Search of Lost Domain Generalization. In ICLR, 2021.

From: NeurIPS.cc/2022/Conference/Paper6146/Authors
title: Reply to R1 (Part 1)
comment: We are thankful for your positive feedback and insightful suggestions. We are happy that you found our paper interesting, relevant, and technically solid; having rigorous experiments and ablations; showing strong empirical gains. We answer your questions here.

---

**Q1**: Minimal detail in main body: There is mention of a formal analysis, and there are several core algorithms that are fully relegated to the appendix; it would seem appropriate and possible to include more in the main body.

**A1**: In this work, we conducted a systematic study of exploiting a zoo of pre-trained models (PTMs) for improving OoD generalization. It involves the PTMs ranking and ensemble with feature selection, each of which is carefully designed and targets one specific problem. Due to a shortage of space, we move the detailed analysis to Appendix. We will, however, add the algorithms and more details to the main paper in the final version.

---

**Q2**: Ablation/examination of pre-trained effects: The authors show strong absolute performance against prior work but don't go into detail on how differences in which pre-trained models were used may have influenced; i.e., are the effects of different/better pre-trained models properly controlled for?


**A2**: 
The influence of PTM has been studied in the literature. Recent works have shown the advantages of using PTMs for improving OoD generalization [1-4]. However, the selection and utilization of a PTM from a model zoo have not been explored yet (Line 29-37). In the absence of any OoD method, we have two options: fine-tuning or using IID ranking metrics. Naive fine-tuning may perform well, but it is prohibitively expensive in terms of the computation required. The second option, IID ranking metrics, does not perform well and does not follow the leave-one-domain-out cross-validation paradigm proposed by Guljrani et al. [5]. In this work, we provide an efficient way to utilize (rank and use) PTMs for OoD generalization. Section 4.1 shows that our method outperforms IID ranking metrics on standard benchmarks (See Figure 2, Figure 3, and Table 1.). 

The strong performance against prior work shows that our method can exploit the power of the PTM zoo (Section 4.2).
From Table 2, We can draw the following conclusions : (i) Using a good PTM zoo can surpass single-model SOTA results; (ii) Our method fully exploit the power of a PTM zoo. In the manuscript, we do not control for the impact of better PTMs, since using a zoo with good PTMs does not lead to unfair comparisons.
(1) Given a zoo of PTMs, our method aims to exploit the power of the zoo for OoD generalization.
(2) Our method can be regarded as a complement to other OoD algorithms. After selecting the top-ranked models, different OoD algorithms can be adapted to fine-tune the models.
(3) We built a model zoo with 35 PTMs. We can further increase the power of the model zoo by adding more PTMs. This does not affect the purpose of our experiments.

To control the influence of PTMs, we performed an ablation study with only ResNet-50 models. However, these models are trained differently (e.g., adversarially trained models). For this experiment, we controlled the architecture (e.g., ResNet-50 only) and OoD training algorithm (e.g., ERM only). We compared the best-selected model with a simple standard baseline (PyTorch ResNet used by most OoD papers) in the following table. 

| Dataset | PACS | VLCS | Office-Home | TerraInc. | DomainNet |
| ------------- | ---- | ---- | ----------- | --------- | --------- |
| Baseline      | 66.9 | 76.7 | 67.7        | 31.0      | 31.1      |
| Ours (Single) | 91.7 | 79.1 | 76.1        | 32.5      | 46.3      |

From: NeurIPS.cc/2022/Conference/Paper6146/Reviewer_z3dR
rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: This paper studies the problem of increasing performance on out of domain (OOD) classification, i.e. generalization/robustness on data distributions different from the training dataset.  In particular, this work focuses on the setting where there are several pre-trained models (PTMs) available.  The goal of the paper is then to use these PTMs in some efficient way to realize high OOD generalization.  The proposed approach breaks this into two parts: (1) ranking the PTMs by predicted generalization OOD, and (2) ensembling them.  The authors propose new approaches for both parts, the first using a ranking metric that uses inter-class discriminative capacity and inter-domain stability computed in a leave-one-out x-validation, and (2) using a feature sub-selection method accomplished using an approximate inference EM approach.  The authors rigorously compare and ablate the proposed approach and show strong empirical gains.
strengths_and_weaknesses: Strengths:
- (S1) Interesting and relevant topic: OOD generalization and robustness is always a critical topic, with growing resources and momentum recently; and increasingly there are many pre-trained models publicly accessible which together makes the setting very relevant.
- (S2) Strong empirical results with thorough methodological ablations: The proposed pipeline is somewhat complex but the component pieces and design choices are thoroughly evaluated overall, with very strong empirical results

Weaknesses:
- (W1) Minimal detail in main body: There is mention of a formal analysis, and there are several core algorithms that are fully relegated to the appendix; it would seem appropriate and possible to include more in the main body.
- (W2) Ablation/examination of pre-trained effects: The authors show strong absolute performance against prior work but don't go into detail on how differences in which pre-trained models were used may have influenced; i.e., are the effects of different/better pre-trained models properly controlled for?
- (W3) The benefits of ensembling over just using the best pre-trained model selected seem fairly minimal
questions: N/A
limitations: N/A
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper6146/Reviewer_SbCP
rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: Naive fine-tuning requires high computational cost for choosing the best performing model on the target task, and the best performing model may not perform well on unseen domains. A simple model also cannot fully utilize the knowledge presented in different PTMs. Motivated by such observations, the paper proposed a ranking metric to inspect the generalization ability of the PTMs on unseen data through quantifying the inter-class discriminability and inter-domain stability. They also propose an EM approach to select the informative features and discard the noisy features. Experimental results show that the proposed approach outperforms the previous SOTA methods with high efficiency.
strengths_and_weaknesses: Strength
1. The paper is well written and easy to follow.
2. The main contributions of the paper consist of a ranking metric of inspecting the generalization-ability of PTMs and a feature selection approach.
3. The paper conducted extensive experiments (35 PTMs and 7 datasets), and the experimental results outperforms the previous best approaches.


Weakness
1. Regarding the experiment results shown in Figure 2, the proposed ZooD approach does not show much better performance when compared to LogME, except on Terralncognita. However, the ZooD requires a linear model with Gaussian error for evaluation. 
2. For the experiment results in Table 2, the experiment result of the Single model outperforms the previous SOTA by a large margin. However, the Single model here indicates a fine-tuning model that is selected from the PTMs.  I am curious why such a baseline is not included in the table. It seems unfair to compare a selected top-1models from a zoo of 35 models with a single model on the 5 datasets. 
3. Regarding the ensemble results in Table 2, do you have the performance comparisons of the Ensemble approach with the LogME approach? 
4. Regarding the computational efficiency, the paper compared the proposed ZooD with brute-force finetuning. When comparing with a simple ensemble, such as LogME, does ZooD still have advantages?

questions: In Line 192, do you refer to the wrong equation?
In Table 2, Feature selection contributes more to the performance improvement compared to model Ensemble. What if the feature selection approach is directly applied on the single model? 

Line 48, there should be a full stop before “Given…’.

limitations: NA
ethics_flag: No
soundness: 3 good
presentation: 4 excellent
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper6146/Reviewer_gvF7
rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: This paper proposes a pre-trained model ranking metric and an ensemble method to improve the generalization on target out-of-distribution tasks. The proposed metric evaluates inter-class discrimination and inter-domain stability in a cross-domain validation manner across a set of domains, so that the model would not fixate on domain-specific features. Besides, authors propose a variational EM algorithm to reduce noise of ensembled features. Extensive experiments of 35 PTMs on six datasets show the effectiveness of the method.
strengths_and_weaknesses: Strengths:
(1) The proposed method is technically solid and reasonable.
(2) Experiments are comprehensive and convincing.
(3) The proposed method significantly outperforms SOTA baselines on most datasets.

Weaknesses:
(1) The complexity of computing proposed metric on m domains is O(m^2), which will be inefficient when there are a large number of domains. Although it is still much faster than brute-force finetuning, how does it compare with other non-finetuning methods?
(2) Authors better put the results of finetuning the whole network in table 2 to improve readability.
(3) It would be better if authors report the finetuning results of the PTM selected by the IID ranking baselines (H-Score, LogME, LEEP and NCE) to better compare the ranking methods.

questions: (1) Why IID ranking baselines perform so bad on TerraIncognita? Is fine-tuning top linear classifiers a good metric on this dataset?
(2) ZooD uses a large set of PTMs including the newly proposed Transformer-based models, while the baselines (e.g., Gulrajani and Lopez-Paz [18]) use ResNet-50. My main concern is that the comparisons may be unfair due to differences in the performance of different architectures. So, do the different network architectures affect the results?

limitations: yes
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 2 fair