Original: SIMPLE: SPECIALIZED MODEL - SAMPLE MATCHING FOR DOMAIN GENERALIZATION
Revised: SIMPLE: SPECIALIZED MODEL - SAMPLE MATCHING FOR DOMAIN GENERALIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In domain generalization (DG), most existing methods aspire to fine-tune a specific pretrained model through novel DG algorithms.
Revised: In domain generalization (DG), most existing methods aspire to fine-tune a specific pretrained model through novel DG algorithms.

Original: In this paper, we propose an alternative direction, i.e., to efficiently leverage a pool of pretrained models without fine-tuning.
Revised: In this paper, we propose an alternative direction, i.e., to efficiently leverage a pool of pretrained models without fine-tuning.

Original: Through extensive empirical and theoretical evidence, we demonstrate that (1) pretrained models have possessed generalization to some extent while there is no single best pretrained model across all distribution shifts, and (2) out-of-distribution (OOD) generalization error depends on the fitness between the pretrained model and unseen test distributions.
Revised: Through extensive empirical and theoretical evidence, we demonstrate that (1) pretrained models have possessed generalization to some extent while there is no single best pretrained model across all distribution shifts, and (2) out-of-distribution (OOD) generalization error depends on the fitness between the pretrained model and unseen test distributions.

Original: This analysis motivates us to incorporate diverse pretrained models and to dispatch the best matched models for each OOD sample by means of recommendation techniques.
Revised: This analysis motivates us to incorporate diverse pretrained models and to dispatch the best matched models for each OOD sample by means of recommendation techniques.

Original: To this end, we propose SIMPLE, a specialized model-sample matching method for domain generalization.
Revised: To this end, we propose SIMPLE, a specialized model-sample matching method for domain generalization.

Original: First, the predictions of pretrained models are adapted to the target domain by a linear label space transformation.
Revised: First, the predictions of pretrained models are adapted to the target domain by a linear label space transformation.

Original: A matching network aware of model specialty is then proposed to dynamically recommend proper pretrained models to predict each test sample.
Revised: A matching network aware of model specialty is then proposed to dynamically recommend proper pretrained models to predict each test sample.

Original: The experiments on DomainBed show that our method achieves significant performance improvements (up to 12.2% for individual dataset and 3.9% on average) compared to state-of-the-art (SOTA) methods and further achieves 6.1% gain via enlarging the pretrained model pool.
Revised: The experiments on DomainBed show that our method achieves significant performance improvements (up to 12.2% for individual dataset and 3.9% on average) compared to state-of-the-art (SOTA) methods and further achieves 6.1% gain via enlarging the pretrained model pool.

Original: Moreover, our method is highly efficient and achieves more than 1000 × training speedup compared to the conventional DG methods with fine-tuning a pretrained model.
Revised: Moreover, our method is highly efficient and achieves more than 1000 × training speedup compared to the conventional DG methods with fine-tuning a pretrained model.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Distribution shift is a common problem in real-world applications, which breaks the independent and identically distributional (i.i.d.) assumption of machine learning algorithms (Wang et al., 2022).
Revised: Distribution shift is a common problem in real-world applications, which breaks the independent and identically distributional (i.i.d.) assumption of machine learning algorithms (Wang et al., 2022).

Original: Mismatches between training and test distributions, which are quite common in reality, can largely deteriorate model performance and make machine learning models infeasible for practical applications (Gonz ´ alez & Abu-Mostafa, 2015).
Revised: Mismatches between training and test distributions, which are quite common in reality, can largely deteriorate model performance and make machine learning models infeasible for practical applications (Gonz ´ alez & Abu-Mostafa, 2015).

Original: Therefore, enhancing the generalization ability of models has attracted increasing attention (Cha et al., 2021; Zhang et al., 2022).
Revised: Therefore, enhancing the generalization ability of models has attracted increasing attention (Cha et al., 2021; Zhang et al., 2022).

Original: For its practical significance, various methods have been proposed, e.g., domain alignment (Ganin et al., 2016;
Revised: For its practical significance, various methods have been proposed, e.g., domain alignment (Ganin et al., 2016;

Original: Gong et al., 2019; Arjovsky et al., 2019), meta-learning (Finn et al., 2017; Dou et al., 2019; Du et al., 2020), and ensemble learning (Mancini et al., 2018; Cha et al., 2021; Arpit et al., 2021).
Revised: Gong et al., 2019; Arjovsky et al., 2019), meta-learning (Finn et al., 2017; Dou et al., 2019; Du et al., 2020), and ensemble learning (Mancini et al., 2018; Cha et al., 2021; Arpit et al., 2021).

Original: A common approach to verify the effectiveness of DG algorithms is to fine-tune a pretrained ResNet (He et al., 2016) model with these algorithms (Gulrajani & Lopez-Paz, 2020).
Revised: A common approach to verify the effectiveness of DG algorithms is to fine-tune a pretrained ResNet (He et al., 2016) model with these algorithms (Gulrajani & Lopez-Paz, 2020).

Original: It has demonstrated that these algorithms improve upon empirical risk minimization (ERM) baseline on
Revised: It has demonstrated that these algorithms improve upon empirical risk minimization (ERM) baseline on

Original: ResNet-50 backbone (Arpit et al., 2021; Wiles et al., 2021).
Revised: ResNet-50 backbone (Arpit et al., 2021; Wiles et al., 2021).

Original: Meanwhile, recent studies show that the neural architectures and pretraining methods have a significant impact on the robustness of models to distribution shifts (Radford et al., 2021; Wiles et al., 2021).
Revised: Meanwhile, recent studies show that the neural architectures and pretraining methods have a significant impact on the robustness of models to distribution shifts (Radford et al., 2021; Wiles et al., 2021).

Original: For example, vision transformers are more robust to texture and style shifts compared with ResNet-based models (Zhang et al., 2022), which are instead superior to transformer-based models on dense image classification tasks (Liu et al., 2022).
Revised: For example, vision transformers are more robust to texture and style shifts compared with ResNet-based models (Zhang et al., 2022), which are instead superior to transformer-based models on dense image classification tasks (Liu et al., 2022).

Original: In terms of pretraining, using pretraining datasets other than ImageNet-1k improves the generalization performance in one test domain, yet causes a degradation in another domain (Kim et al., 2022).
Revised: In terms of pretraining, using pretraining datasets other than ImageNet-1k improves the generalization performance in one test domain, yet causes a degradation in another domain (Kim et al., 2022).

Original: These findings are in line with the No Free Lunch (NFL) Theorem (Wolpert, 1996), which suggests that no single model can always perform better than any other model without having substantive information about the targeted problem.
Revised: These findings are in line with the No Free Lunch (NFL) Theorem (Wolpert, 1996), which suggests that no single model can always perform better than any other model without having substantive information about the targeted problem.

Original: In DG, we usually have very limited information about the test domain, so we are more likely to encounter the above challenge.
Revised: In DG, we usually have very limited information about the test domain, so we are more likely to encounter the above challenge.

Original: Inspired by these attempts, in this paper
Revised: Inspired by these attempts, in this paper

Original: , we conduct a fine-grained study on the relationship between pretrained models and distribution shifts.
Revised: , we conduct a fine-grained study on the relationship between pretrained models and distribution shifts

Original: From both empirical and theoretical envidence, we show that no free lunch in terms of pretraining for domain generalization , i.e., there is no single best pretrained model across shifting test distributions.
Revised: From both empirical and theoretical envidence, we show that no free lunch in terms of pretraining for domain generalization , i.e., there is no single best pretrained model across shifting test distributions.

Original: Specifically,  pretrained models with different network architectures, pretraining datasets and learning objectives are compared for their generalization performance under different distributional shifts.
Revised: Specifically, 283 pretrained models with different network architectures, pretraining datasets and learning objectives are compared for their generalization performance under different distributional shifts.

Original: The results reveal that the pretrained mod els without fine-tuning generalize well to some unseen domains, but none of these models dominate in all unseen distributions.
Revised: The results reveal that the pretrained models without fine-tuning generalize well to some unseen domains , but none of these models dominate in all unseen distributions.

Original: Furthermore, the theoretical analysis indicates that OOD generalization error is determined by the fitness between model (varying w.r.t.
Revised: Furthermore, the theoretical analysis indicates that OOD generalization error is determined by the fitness between model (varying w. r .t

Original: the network architectures and model weights) and test distribution.
Revised: the network architectures and model weights) and test distribution.

Original: For any network architecture with fixed training distributions, such as pretrained models (Iandola et al., 2014; He et al., 2016; 2021a), it is always possible to find a beneficial or detrimental test distribution with a small or large generalization error.
Revised: For any network architecture with fixed training distributions, such as pretrained models (Iandola et al., 2014; He et al., 2016; 2021a), it is always possible to find a beneficial or detrimental test distribution with a small or large generalization error.

Original: Motivated by these findings, we propose an alternative DG paradigm that leverages pretrained models with different network architectures and shifting training distributions, upon which we match the most suitable pretrained models for each OOD sample.
Revised: Motivated by these findings, we propose an alternative DG paradigm that leverages pretrained models with different network architectures and shifting training distributions, upon which we match the most suitable pretrained models for each OOD sample.

Original: As shown in Figure 3, our paradigm ( s pec i alized m odel-sam ple matching for domain generalization, SIMPLE)
Revised: As shown in Figure 3, our paradigm ( s pec i alized m odel-sam ple matching for domain generalization, SIMPLE)

Original: first adopts a simple label adapter that projects the label space of the pretrained domain  to that of the target domain, where the adapter is shared by pretrained models from the same label space.
Revised: first adopts a simple label adapter that projects the label space of the pretraining domain 1 to that of unseen domains 2 , where the adapter is shared by pretrained models from the same pretraining domain.

Original: Then, a matching network, which is aware of model specialty, selects a set of proper pretrained models and aggregates them together to conduct the prediction for each OOD sample.
Revised: Then, a matching network, which is aware of model specialty, selects a set of proper pretrained models and aggregates them together to conduct the prediction for each OOD sample.

Original: Notably, this promising alternative exhibits significant performance improvements, averaging 3.9% over the existing SOTA results, with gains of up to 12.2% on single datasets, and a significant increase in training efficiency.
Revised: Notably, this promising alternative exhibits significant performance improvements, averaging 3.9% over the existing SOTA results, with gains of up to 12.2% on single datasets, and a significant increase in training efficiency.

Original: To summarize, this work has made the following contributions:
Revised: To summarize, this work has made the following contributions:

Original: • We theoretically and empirically analyze the generalization of pretrained models on shifting unseen test distribution, revealing no free lunch hypothesis exists that motivates our solution of modelsample matching.
Revised: • We theoretically and empirically analyze the generalization of pretrained models on shifting unseen test distribution, revealing no free lunch hypothesis exists that motivates our solution of modelsample matching.

Original: • In complementary to traditional DG solutions, we propose a novel DG paradigm which directly leverages pretrained models without fine-tuning, and it has significantly improved the DG performance in the mainstream benchmark upon other strong baselines.
Revised: • In complementary to traditional DG solutions, we propose a novel DG paradigm which directly leverages pretrained models without fine-tuning, and it has significantly improved the DG performance in the mainstream benchmark upon other strong baselines.

Original: • Besides the performance gain, our method is even more efficient since it does not follow the common fine-tuning approach, shedding new light on using pretrained models in DG tasks.
Revised: • Besides the performance gain, our method is even more efficient since it does not follow the common fine-tuning approach, shedding new light on using pretrained models in DG tasks.

Original:  N O F REE L UNCH IN P RETRAINING FOR D OMAIN G ENERALIZATION
Revised:  N O F REE L UNCH IN P RETRAINING FOR D OMAIN G ENERALIZATION

Original: In this section, we investigate if there exists free lunch in pretraining for DG, that is, whether we can find one single best pretrained model that generalizes across all distributional shifts.
Revised: In this section, we investigate if there exists free lunch in pretraining for DG, that is, whether we can find one single best pretrained model that generalizes across all distributional shifts.

Original: To this end, we first conduct an empirical analysis on the generalization ability of pretrained models over shifting distributions in Section 2.1, followed by a theoretical analysis in Section 2.2.
Revised: To this end, we first conduct an empirical analysis on the generalization ability of pretrained models over shifting distributions in Section 2.1, followed by a theoretical analysis in Section 2.2.

Original: 2. G ENERALIZABILITY ANALYSIS OF THE PRETRAINED MODELS
Revised: 2. G ENERALIZABILITY ANALYSIS OF THE PRETRAINED MODELS

Original: We here analyze the generalization ability possessed by different pretrained models.
Revised: We here analyze the generalization ability possessed by different pretrained models.

Original: Note that existing DG methods generally adopt a specific ImageNet-pretrained model (e.g, ResNet-50), which has been shown not sufficient for generalization (Kumar et al., 2021; Kim et al., 2022).
Revised: Note that existing DG methods generally adopt a specific ImageNet-pretrained model (e.g, ResNet-50), which has been shown not sufficient for generalization (Kumar et al., 2021; Kim et al., 2022).

Original: Thus, for a comprehensive analysis, we first incorporate 283 diverse pretrained models composed of diverse combinations of network architectures, pretraining datasets, objectives, and algorithms.
Revised: Thus, for a comprehensive analysis, we first incorporate 283 diverse pretrained models composed of diverse combinations of network architectures, pretraining datasets, objectives, and algorithms.

Original: Detailed information on all these models and more experimental settings are in Appendix A.4.
Revised: Detailed information on all these models and more experimental settings are in Appendix A.4.

Original: For the efficient adaptation of pretrained models from pretraining domains to target domains, we propose to train only a label space adapter upon the pretrained models without modifying any pretrained model parameters, which is remarkably lightweight and will be elaborated detailedly in Section 3.2.
Revised: For the efficient adaptation of pretrained models from pretraining domains to unseen domains, we propose to train only a label space adapter upon the pretrained models without modifying any pretrained model parameters, which is remarkably lightweight and will be elaborated detailedly in Section 3.2.

Original: Takeaway 1: Pretrained models possess decent generalization ability for some OOD samples.
Revised: Takeaway 1: Pretrained models possess decent generalization ability for some OOD samples.

Original: By grafting such a lightweight label space adapter, we find that the generalization performance of the pretrained model is already promising in some cases.
Revised: By grafting such a lightweight label space adapter, we find that the generalization performance of the pretrained model is already promising in some cases.

Original: As a concrete example, given a fixed
Revised: As a concrete example, given a fixed

Original: DenseNet-121 model (Iandola et al., 2014) pretrained on ImageNet, we employ an adapter that converts its prediction of the original ImageNet label space to that of OfficeHome (a dataset of
Revised: DenseNet-121 model (Iandola et al., 2014) pretrained on ImageNet, we employ an adapter that converts its prediction of the original ImageNet label space to that of OfficeHome (a dataset of

Original: DG benchmark) (Venkateswara et al., 2017), with the adapter training on source domains.
Revised: DG benchmark) (Venkateswara et al., 2017), with the adapter training on source domains.

Original: This combination achieves an average accuracy of 78.3% on target domains, which is 5.9% higher than
Revised: This combination achieves an average accuracy of 78.3% on target domains, which is 5.9% higher than

Original: SOTA as detailed in Section 4.3.
Revised: SOTA as detailed in Section 4.3.

Original: Takeaway 2: No dominant pretrained models across unseen domains.
Revised: Takeaway 2: No dominant pretrained models across unseen domains.

Original: Though pretrained models have possessed some generalization ability in some cases, however, their generalization performance relates to the specific unseen distributions.
Revised: Though pretrained models have possessed some generalization ability in some cases, however, their generalization performance relates to the specific unseen distributions.

Original: The left part of Figure 1 shows the relative performance of the pretrained models evaluated in different domains, with label adapter.
Revised: The left part of Figure 1 shows the relative performance of the pretrained models evaluated in different domains, with label adapter.

Original: Each domain represents a different data distribution.
Revised: Each domain represents a different data distribution.

Original: As can be seen, pretrained models vary greatly in performance on different test domains, without any single model being dominant in all domains.
Revised: As can be seen, pretrained models vary greatly in performance on different test domains, without any single model being dominant in all domains.

Original: Takeaway 3: Pretrained models exhibit more diverse performance at finer-grained levels.
Revised: Takeaway 3: Pretrained models exhibit more diverse performance at finer-grained levels.

Original: We further examine whether performance divergence also exists at a finer-grained level, such as classlevel.
Revised: We further examine whether performance divergence also exists at a finer-grained level, such as classlevel.

Original: The right part of Figure 1 presents the relative model performance on 10 classes in TerraIncognita dataset.
Revised: The right part of Figure 1 presents the relative model performance on 10 classes in TerraIncognita dataset.

Original: Similar to that at domain-level, varying model performance among different classes is also noticeable.
Revised: Similar to that at domain-level, varying model performance among different classes is also noticeable.

Original: Moreover, there exists a more pronounced divergence in model performance at the finer class level, as evidenced in detail in Appendix A.4.
Revised: Moreover, there exists a more pronounced divergence in model performance at the finer class level, as evidenced in detail in Appendix A.4.

Original: It supports the necessity of incorporating pretrained models and being aware of their specialty at a fine-grained level, which motivates us to investigate and exploit the matching metric between models and test samples.
Revised: It supports the necessity of incorporating pretrained models and being aware of their specialty at a fine-grained level, which motivates us to investigate and exploit the matching metric between models and test samples.

Original: 2. T HEORETICAL ANALYSIS OF N O F REE L UNCH FOR DG
Revised: 2. T HEORETICAL ANALYSIS OF N O F REE L UNCH FOR DG

Original: This section provides theoretical analysis to support the findings in Section 2.1.
Revised: This section provides theoretical analysis to support the findings in Section 2.1.

Original: To alleviate the generalization difficulties associated with the train-test mismatch, mainstream DG methods seek domain-invariant features that can be generalized beyond the training distribution.
Revised: To alleviate the generalization difficulties associated with the train-test mismatch, mainstream DG methods seek domain-invariant features that can be generalized beyond the training distribution.

Original: However, there are several issues to consider.
Revised: However, there are several issues to consider.

Original: First, domain-invariant features learned from source domains can still be biased towards source domains and thus have limited performance on unseen domains (Cha et al., 2022).
Revised: First, domain-invariant features learned from source domains can still be biased towards source domains and thus have limited performance on unseen domains (Cha et al., 2022).

Original: Second, domain-specific information has also been found to enhance DG, as it may be closely related to the sample labels that help generalize in certain unseen domains (Bui et al., 2021).
Revised: Second, domain-specific information has also been found to enhance DG, as it may be closely related to the sample labels that help generalize in certain unseen domains (Bui et al., 2021).

Original: These observations suggest that existing DG methods may not guarantee whether a model can generalize across distinct unseen target domains, the reason lies in the limitation of our knowledge of unseen domains, which is in line with NFL theorem.
Revised: These observations suggest that existing DG methods may not guarantee whether a model can generalize across distinct unseen target domains, the reason lies in the limitation of our knowledge of unseen domains, which is in line with NFL theorem.

Original: The analysis of the OOD generalization of kernel methods also points out the relevant insight that shift in test distribution may help or hurt generalization (Canatar et al., 2021).
Revised: The analysis of the OOD generalization of kernel methods also points out the relevant insight that shift in test distribution may help or hurt generalization (Canatar et al., 2021).

Original: Theorem 1.
Revised: Theorem 1.

Original: (Informal; OOD Kernel Generalization;Canatar et al.
Revised: (Informal; OOD Kernel Generalization;Canatar et al.

Original: Given the kernel matrix
Revised: Given the kernel matrix

Original: K ( x, x ′ ) with Mercer decomposition K ( x, x ′ ) =
Revised: K ( x, x ′ ) with Mercer decomposition K ( x, x ′ ) =

Original: Φ ( x ) T ΛΦ ( x ′ ) .
Revised: Φ ( x ) T ΛΦ ( x ′ ) .

Original: Suppose the training data is i.i.d.
Revised: Suppose the training data is i.i.d.

Original: generated from the distribution p ( x ) , the target function is given by y = ¯ a T Φ ( x ) , and the training loss is kernel regression with ERM.
Revised: generated from the distribution p ( x ) , the target function is given by y = ¯ a T Φ ( x ) , and the training loss is kernel regression with ERM.

Original: The generalization error on distribution ˜ p ( x ) is given by E g
Revised: The generalization error on distribution ˜ p ( x ) is given by E g

Original: =
Revised: =

Original: E matched g
Revised: E matched g

Original: + κ ¯ a T ( P Λ + κ I ) − 1 O ′
Revised: + κ ¯ a T ( P Λ + κ I ) − 1 O ′

Original: ( P Λ + κ I ) − 1 ¯ a , where O ij = (cid:82) dx ˜ p
Revised: ( P Λ + κ I ) − 1 ¯ a , where O ij = (cid:82) dx ˜ p

Original: ( x ) ϕ i ( x ) ϕ j ( x ) and
Revised: ( x ) ϕ i ( x ) ϕ j ( x ) and

Original: O ′ =
Revised: O ′ =

Original: O − 1 − γ ′ 1 − γ I .
Revised: O − 1 − γ ′ 1 − γ I .

Original: Here E matched g is the generalization error when the training distribution and test distribution are matched, P is the number of training samples, κ, γ are constants depend on Λ , and γ ′ is a constant depend on Λ and O .
Revised: Here E matched g is the generalization error when the training distribution and test distribution are matched (i.e., in-distribution error), P is the number of training samples, κ, γ are constants depend on Λ , and γ ′ is a constant depend on Λ and O .

Original: Theorem 1 will be detailed in Appendix A.5.
Revised: Theorem 1 will be detailed in Appendix A.5.

Original: Remark 1.
Revised: Remark 1.

Original: (Interpretations of Theorem 1) Theorem 1 shows that E g is in the form E g =
Revised: (Interpretations of Theorem 1) Theorem 1 shows that E g is in the form E g =

Original: E matched g + v O ′ v , where the second term is due to distribution shift.
Revised: E matched g + v O ′ v , where the second term is due to distribution shift.

Original: The theorem is also applicable to large models as studies have demonstrated that infinitely wide neural networks are equivalent to neural tangent kernels (Jacot et al., 2018).
Revised: Note that, if a pretrained model is kept frozen to serve as a static feature extractor for the subsequent trainable linear classifier, the pretrained model acts as a kernel and thus Theorem 1 applies.

Original: Remark 2.
Revised: Remark 2.

Original: (Generalization depends on the fitness between model and test distributions)
Revised: (Generalization depends on the fitness between model and test distributions)

Original: The eigenvalue of matrix O depends on the alignment between the test distribution ˜ p ( x ) and the kernel basis Φ .
Revised: The eigenvalue of matrix O depends on the alignment between the test distribution ˜ p ( x ) and the kernel basis Φ which is determined by the model (network architecture and model weights).

Original: The matrix O ′ may have negative eigenvalues when ˜ p ( x ) and Φ are matched well and OOD generalization is even better than i.i.d.
Revised: The matrix O ′ may have negative eigenvalues when ˜ p ( x ) and Φ are matched well and OOD generalization is even better than i.i.d.

Original: generalization under such circumstances.
Revised: generalization under such circumstances.

Original: On the contrary, when Φ ( x ) is fixed, ˜ p ( x ) can be adversarially chosen such that the matrix O ′ has large positive eigenvalues and thus the network fails to generalization to OOD data.
Revised: On the contrary, when Φ ( x ) is fixed, ˜ p ( x ) can be adversarially chosen such that the matrix O ′ has large positive eigenvalues and thus the network fails to generalization to OOD data.

Original: Remark 3.
Revised: Remark 3.

Original: (No free lunch for single model in DG)
Revised: (No free lunch for single model in DG)

Original: A major focus of DG is tackling covariate shift, where ˜ p ( x ) can be set arbitrarily as long as p ( y | x ) =
Revised: A major focus of DG is tackling covariate shift, where ˜ p ( x ) can be set arbitrarily as long as p ( y | x ) =

Original: ˜ p ( y | x ) .
Revised: ˜ p ( y | x ) .

Original: Under covariate shift, no single pretrained model will outperform other models for all ˜ p ( x ) , implying no free lunch theorem in DG.
Revised: Under covariate shift, no single pretrained model will outperform other models for all ˜ p ( x ) , implying no free lunch theorem in DG.

Original: The theoretical analysis above raises a natural question: should not the focus be more on matching the pretrained model and testing distributions based on their fitness?
Revised: The theoretical analysis above raises a natural question: should not the focus be more on matching the pretrained model and testing distributions based on their fitness?

Original: Given that pretrained models with fixed network architectures and training distributions always face beneficial or detrimental test distributions, we suggest incorporating more pretrained models to create diverse network architectures and shifting training distributions, to facilitate generalization.
Revised: Given that pretrained models with fixed network architectures and training distributions always face beneficial or detrimental test distributions, we suggest incorporating more pretrained models to create diverse network architectures and shifting training distributions, to facilitate generalization.

Original: Depending on the fitness of the pretrained models to the target distributions, it is then possible to match OOD samples to appropriate models, thus bypassing the issue of using a single model indicated by the NFL theorem.
Revised: Depending on the fitness of the pretrained models to the target distributions, it is then possible to match OOD samples to appropriate models, thus bypassing the issue of using a single model indicated by the NFL theorem.

Original:  T HE P ROPOSED F RAMEWORK FOR D OMAIN G ENERALIZATION
Revised:  T HE P ROPOSED F RAMEWORK FOR D OMAIN G ENERALIZATION

Original: In this section, we present a novel framework, namely s pec i alized m odel-sam ple matching for domain generalization (SIMPLE), that reformulates DG as a matching problem, in light of the analysis in Section 2.
Revised: In this section, we present a novel framework, namely s pec i alized m odel-sam ple matching for domain generalization (SIMPLE), that reformulates DG as a matching problem, in light of the analysis in Section 2.

Original: We first introduce the overall framework in Section 3.1.
Revised: We first introduce the overall framework in Section 3.1.

Original: Then, we elaborate the design of specialty-aware model-sample matching and ensemble in Section 3.2, followed by the learning algorithm in Section 3.3.
Revised: Then, we elaborate the design of specialty-aware model-sample matching and ensemble in Section 3.2, followed by the learning algorithm in Section 3.3.

Original: 3. T HE OVERALL FRAMEWORK
Revised: 3. T HE OVERALL FRAMEWORK

Original: This section provides an overview of the SIMPLE framework, as shown in Figure 3.
Revised: This section provides an overview of the SIMPLE framework, as shown in Figure 3.

Original: Following our analysis in Section 2, one single pretrained model is not sufficient to accommodate diverse
Revised: Following our analysis in Section 2, one single pretrained model is not sufficient to accommodate diverse

Original: OOD samples.
Revised: OOD samples.

Original: From an intuitive view, it appears that more models need to be incorporated and selected to solve the problem.
Revised: From an intuitive view, it appears that more models need to be incorporated and selected to solve the problem.

Original: Analogous to recommending items (models) to users (samples) from the vast item set in recommender system, we formulate DG as a model-sample matching problem, with a model pool containing various models and a model dispatcher responsible for assigning these models to OOD samples appropriately.
Revised: Analogous to recommending items (models) to users (samples) from the vast item set in recommender system, we formulate DG as a model-sample matching problem, with a model pool containing various models and a model dispatcher responsible for assigning these models to OOD samples appropriately.

Original: Preliminaries.
Revised: Preliminaries.

Original: DG aims to tackle the shift of data distribution among different domains by transferring knowledge from seen to unseen domains.
Revised: DG aims to tackle the shift of data distribution among different domains by transferring knowledge from seen to unseen domains.

Original: Unlike domain adaptation, samples from unseen target domain(s) are inaccessible in DG.
Revised: Unlike domain adaptation, samples from unseen target domain(s) are inaccessible in DG.

Original: For a domain, its input and label space can be denoted as
Revised: For a domain, its input and label space can be denoted as

Original: X ∈ R d and Y ∈ R C , respectively, where d is the dimension of input and C is the number of classes in Y ; And its samples are observed constructing a dataset D
Revised: X ∈ R d and Y ∈ R C , respectively, where d is the dimension of input and C is the number of classes in Y ; And its samples are observed constructing a dataset D

Original: = { ( x i , y i ) }
Revised: = { ( x i , y i ) }

Original: Ni =1 with sample number N .
Revised: Ni =1 with sample number N .

Original: Consider that we have S source domains D s =
Revised: Consider that we have S source domains D s =

Original: { D 1 , . . .
Revised: { D 1 , . . .

Original: , D S } and T target domains
Revised: , D S } and T target domains

Original: D t = { D 1 , . . .
Revised: D t = { D 1 , . . .

Original: , D T } that share the same label space but with different joint distributions on X × Y .
Revised: , D T } that share the same label space but with different joint distributions on X × Y .

Original: Pretrained model pool.
Revised: Pretrained model pool.

Original: As discussed in Section 2.2, the OOD generalization error depends on the fitness between the pretrained model and the test distribution, which is unknown in DG.
Revised: As discussed in Section 2.2, the OOD generalization error depends on the fitness between the pretrained model and the test distribution, which is unknown in DG.

Original: Therefore, building a pool with diverse pretrained models is crucial for DG.
Revised: Therefore, building a pool with diverse pretrained models is crucial for DG.

Original: Note that with abundant pretrained models being released, it is easy to construct a model pool by simply downloading pretrained models from public repositories (Wightman, 2019), without further efforts such as retraining.
Revised: Note that with abundant pretrained models being released, it is easy to construct a model pool by simply downloading pretrained models from public repositories (Wightman, 2019), without further efforts such as retraining.

Original: We collect extensive pretrained models to serve for SIMPLE, as detailed in Appendix A.3.
Revised: We collect extensive pretrained models to serve for SIMPLE, as detailed in Appendix A.3.

Original: We denote a model pool with K models as { f k } Kk =1 , each of which is parameterized as θ k .
Revised: We denote a model pool with K models as { f k } K k =1 , each of which is parameterized as θ k .

Original: Label space adapter.
Revised: Label space adapter.

Original: As the label space of the pretraining domains generally differs fromthat of the  target domain, label space adapters are required to make them consistent.
Revised: As the label space of the pretraining domains generally differs from the one shared by source and target domains, label space adapters are required to make them consistent.

Original: The adapter is a linear mapping from the label space of the original pretraining domains to that of the target domains, which is shared among different pretrained models from the same pretraining domain (e.g., we only use one adapter for all the models trained on ImageNet-1k).
Revised: The adapter is a linear mapping between these two label spaces (pretraining → source/target), which is shared among different pretrained models from the same pretraining domain (e.g., we only use one adapter for all the models trained on ImageNet-1k).

Original: Specifically, given pretrained model f k , we parameterize the adapted model h ψ ( f k ( · ; θ k )) as θ ′ k
Revised: Specifically, given pretrained model f k , we parameterize the adapted model h ψ ( f k ( · ; θ k )) as θ ′ k

Original: =
Revised: =

Original: [ ψ ; θ k ] , where ψ denotes the parameters of the adapter,
Revised: [ ψ ; θ k ] , where ψ denotes the parameters of the adapter,

Original: h ψ ∈
Revised: h ψ ∈

Original: A : R C o
Revised: A : R C o

Original: → R C and C
Revised: → R C and C

Original: o is the dimension of the label space of the original pretraining dataset of f k .
Revised: o is the dimension of the label space of the original pretraining dataset of f k .

Original: Through the label adapter, the output of pretrained model f k can be transformed and adapted to target domains as ˆ y ik = h ψ ( f k ( x i )) , without fine-tuning { θ k } Kk =1 .
Revised: Through the label adapter, the output of pretrained model f k can be transformed and adapted to target domains as ˆ y ik = h ψ ( f k ( x i )) , without fine-tuning { θ k } Kk =1 .

Original: In this way, it can largely reduce the adaptation cost of the pretrained models onto the new domains.
Revised: In this way, it can largely reduce the adaptation cost of the pretrained models onto the new domains.

Original: By the above construction, SIMPLE differs from the existing methods in two aspects.
Revised: By the above construction, SIMPLE differs from the existing methods in two aspects.

Original: First, most existing DG methods strive to fine-tune a specific pretrained model, which cannot generalize well to a variety of unseen domains, as analyzed in Section 2.
Revised: First, most existing DG methods strive to fine-tune a specific pretrained model, which cannot generalize well to a variety of unseen domains, as analyzed in Section 2.

Original: Second, conventional options utilizing the pretrained models are fine-tuning and linear probing, as shown in Figure 2(A) and (B), respectively.
Revised: Second, conventional options utilizing the pretrained models are fine-tuning and linear probing, as shown in Figure 2(A) and (B), respectively.

Original: In detail, fine-tuning the pretrained model is costly and can hurt generalization ability (Wortsman et al., 2022), while linear probing (i.e., replacing the last layer of the pretrained model and retraining that) may achieve better accuracy in OOD scenarios than fine-tuning the whole model (Kumar et al., 2021).
Revised: In detail, fine-tuning the pretrained model is costly and can hurt generalization ability (Wortsman et al., 2022), while linear probing (i.e., replacing the last layer of the pretrained model and retraining that) may achieve better accuracy in OOD scenarios than fine-tuning the whole model (Kumar et al., 2021).

Original: However, the linear probing layer is not transferable across models, making the way either costly when a large number of models need to adapt to new target domains.
Revised: However, the linear probing layer is not transferable across models, making the way either costly when a large number of models need to adapt to new target domains.

Original: Our label space adaption shown in Figure 2(C), instead, is remarkably light since it is shared by all the models from the same pretraining domain, which has also been empirically verified in Section 4.2.
Revised: Our label space adaption shown in Figure 2(C), instead, is remarkably light since it is shared by all the models from the same pretraining domain, which has also been empirically verified in Section 4.2.

Original: [Figure]
Revised: [Figure]

Original: Model dispatcher.
Revised: Model dispatcher.

Original: As shown in Section 2, there is no individual model performing the best across different tasks, thus, appropriate models need to be dispatched to address each specific task.
Revised: As shown in Section 2, there is no individual model performing the best across different tasks, thus, appropriate models need to be dispatched to address each specific task.

Original: We define a model dispatcher g ρ , with parameter ρ , that takes the sample x i as input and determines the weight w k assigned to model f k for the sample x i with (cid:80) Kk = 1 w k = 1 .
Revised: We define a model dispatcher g ρ , with parameter ρ , that takes the sample x i as input and determines the weight w k assigned to model f k for the sample x i with (cid:80) K k =1 w k = 1 .

Original: Here w k represents an estimate of the relative match between the model f k in the model pool and the sample.
Revised: Here w k represents an estimate of the relative match between the model f k in the model pool and the sample.

Original: Based on the constructed model pool and the model dispatcher, the prediction for each test sample is an ensemble of the predictions by the dispatched models.
Revised: Based on the constructed model pool and the model dispatcher, the prediction for each test sample is an ensemble of the predictions by the dispatched models.

Original: The final prediction is given by ˆ y i
Revised: The final prediction is given by ˆ y i

Original: = (cid:80)
Revised: = (cid:80)

Original: Kk =1 [ w k · h ψ ( f k ( x i ))] .
Revised: Kk =1 [ w k · h ψ ( f k ( x i ))] .

Original: Finally, we can define a population loss as E D ( ψ, ρ ) = 1 |D| (cid:80) |D| j =1 E x i ∼ D j [ l (ˆ y i , y i )] over the given domain D .
Revised: Finally, we can define a population loss as E D ( ψ, ρ ) = 1 |D| (cid:80) |D| j =1 E x i ∼ D j [ l (ˆ y i , y i )] over the given domain D .

Original: The objective is to minimize the task-specific loss l (e.g., cross-entropy loss for classification) over both source domains D s and target domains
Revised: The objective is to minimize the task-specific loss l (e.g., cross-entropy loss for classification) over both source domains D s and target domains

Original: D t by only minimizing the empirical risk ˆ E D s ( ψ, ρ ) w.r.t.
Revised: D t by only minimizing the empirical risk ˆ E D s ( ψ, ρ ) w.r.t.

Original: ψ and ρ .
Revised: ψ and ρ .

Original: The performance on the target domains, then, measures the generalization ability.
Revised: The performance on the target domains, then, measures the generalization ability.

Original: 3. S PECIALTY - AWARE M ODEL - SAMPLE M ATCHING AND E NSEMBLE
Revised: 3. S PECIALTY - AWARE M ODEL - SAMPLE M ATCHING AND E NSEMBLE

Original: Following the paradigm introduced in Section 3.1, this section elaborates on the model dispatcher, which consists of a model-sample matching network and a specialty-aware ensemble layer.
Revised: Following the paradigm introduced in Section 3.1, this section elaborates on the model dispatcher, which consists of a model-sample matching network and a specialty-aware ensemble layer.

Original: Model-sample matching.
Revised: Model-sample matching.

Original: To capture the fitness of pretrained models towards OOD samples, a network is proposed to learn the model-sample matching function.
Revised: To capture the fitness of pretrained models towards OOD samples, a network is proposed to learn the model-sample matching function.

Original: We employ a simple recommendation algorithm
Revised: We employ a simple recommendation algorithm

Original: (but not restricted), neural collaborative filtering (NCF)
Revised: (but not restricted), neural collaborative filtering (NCF)

Original: (He et al., 2017), to initiate a proof-of-concept for our idea.
Revised: (He et al., 2017), to initiate [Figure] Input shared ··· ··· ··· ··· 0.

Revised: sample embedding ··· ···

Original: Following NCF, the matching scores m i =
Revised: Prediction adapted outputs learnable model embeddings ··· featureextractor Model-sample matching Specialty [Figure]

Revised: awareensemble layer

Revised: [Figure]

Revised: dispatcher matchingscores

Revised: [Figure]

Revised: a proof-of-concept for our idea.

Revised: Following NCF, the matching scores m i =

Original: [ m i 1 , . . .
Revised: [ m i 1 , . . .

Original: , m iK ] ∈ R K of sample x i and models { f k } Kk =1 are computed on their latent features
Revised: , m iK ] ∈ R K of sample x i and models { f k } Kk =1 are computed on their latent features

Original: c i and c =
Revised: c i and c =

Original: [ c 1 , . .
Revised: [ c 1 , . .

Original: c K ] , through a non-linear function activated multi-layer perceptrons (MLP) as m i
Revised: c K ] , through a non-linear function activated multi-layer perceptrons (MLP) as m i

Original: =
Revised: =

Original: MLP ( c ⊤ i c ) .
Revised: MLP ( c ⊤ i c ) .

Original: Specif-

Original: [Figure]

Original: Input shared ··· ··· ··· ··· 0.

Original: sample embedding ··· ···

Original: Prediction

Original: [Figure]

Original: outputs learnable model embeddings ··· featureextractor

Original: Model-sample matching

Original: Specialty

Original: [Figure] awareensemble layer [Figure] ically, the sample and the model are first embedded and then transformed into a joint latent space.
Revised: Specifically, the sample and the model are first embedded and then transformed into a joint latent space.

Original: The feature extractor of one pretrained model f k 0 from our model pool, fixed as the sample feature extractor, generates sample embedding e i for the sample x i .
Revised: The feature extractor of one pretrained model f k 0 from our model pool, fixed as the sample feature extractor, generates sample embedding e i for the sample x i .

Original: For the embedding of each model f k , we introduce a learnable embedding e k , included in ρ , which is randomly initialized and optimized during training.
Revised: For the embedding of each model f k , we introduce a learnable embedding e k , included in ρ , which is randomly initialized and optimized during training.

Original: Both of them are processed by two non-linear function activated MLP as c i =
Revised: Both of them are processed by two non-linear function activated MLP as c i =

Original: MLP ( MLP ( e i )) and
Revised: MLP ( MLP ( e i )) and

Original: c k =
Revised: c k =

Original: MLP ( MLP ( e k )) , to map into the same space for matching scoring.
Revised: MLP ( MLP ( e k )) , to map into the same space for matching scoring.

Original: Specialty-aware ensemble.
Revised: Specialty-aware ensemble.

Original: After constructing a series of ranked models based on the matching metric, the subsequent goal is to conduct prediction by selecting the most proper models.
Revised: After constructing a series of ranked models based on the matching metric, the subsequent goal is to conduct prediction by selecting the most proper models.

Original: We argue that individual model prediction may not cover most of the target distribution, thus, instead of utilizing only the top-1 matched pretrained model, we propose to apply specialty-aware model ensemble to derive the final prediction for the specific sample.
Revised: We argue that individual model prediction may not cover most of the target distribution, thus, instead of utilizing only the top-1 matched pretrained model, we propose to apply specialty-aware model ensemble to derive the final prediction for the specific sample.

Original: It is also considered that ensemble has shown improved robustness (Lakshminarayanan et al., 2016) and model specialty in various tasks also plays a key role in prediction (Gontijo-Lopes et al., 2021), which will be further elaborated in Section 3.3.
Revised: It is also considered that ensemble has shown improved robustness (Lakshminarayanan et al., 2016) and model specialty in various tasks also plays a key role in prediction (Gontijo-Lopes et al., 2021), which will be further elaborated in Section 3.3.

Original: Specifically, we normalize matching scores by softmax function Softmax ( z ) j =
Revised: Specifically, we normalize matching scores by softmax function Softmax ( z ) j =

Original: [Equation]
Revised: [Equation]

Original: to highlight the relative competition among the pretrained models.
Revised: to highlight the relative competition among the pretrained models.

Original: That is, the ensemble weights w i =
Revised: That is, the ensemble weights w i =

Original: [ w 1 , . . .
Revised: [ w 1 , . . .

Original: , w k , . .
Revised: , w k , . .

Original: . , w K ] ∈ R K , is computed as w i =
Revised: . , w K ] ∈ R K , is computed as w i =

Original: Softmax ( m i ) .
Revised: Softmax ( m i ) .

Original: To save the inference time, we further select models with the top- ˜ k
Revised: To save the inference time, we further select models with the top- ˜ k

Original: ( ˜ k < K ) matching scores, before their inference for the given sample.
Revised: ( ˜ k < K ) matching scores, before their inference for the given sample.

Original: In this paper, we set ˜ k
Revised: In this paper, we set ˜ k

Original: = 6 and the sensitivity analysis on ˜ k will be shown in Section 4.5.
Revised: = 6 and the sensitivity analysis on ˜ k will be shown in Section 4.5.

Original: The overall inference cost is small as illustrated in Section 4.2.
Revised: The overall inference cost is small as illustrated in Section 4.2.

Original: 3.  O BJECTIVE AND LEARNING FOR SPECIALIZED MODEL - SAMPLE MATCHING
Revised:  O BJECTIVE AND LEARNING FOR SPECIALIZED MODEL - SAMPLE MATCHING

Original: Loss for ensemble learning.
Revised: Loss for ensemble learning.

Original: The classification loss L ens ( ψ, ρ ) =
Revised: The classification loss L ens ( ψ, ρ ) =

Original: l (ˆ y i , y i ) is optimized for the likelihood of final ensemble output, to update both the matching network and the adapter.
Revised: l (ˆ y i , y i ) is optimized for the likelihood of final ensemble output, to update both the matching network and the adapter.

Original: Loss for label space adapter learning.
Revised: Loss for label space adapter learning.

Original: To train the general label space adapter h ψ for all pretrained models, we incorporate the weighted classification losses of adapted predictions of pretrained models to update the shared adapter, which is defined as follows:
Revised: To train the general label space adapter h ψ for all pretrained models, we incorporate the weighted classification losses of adapted predictions of pretrained models to update the shared adapter, which is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: Loss for model specialty learning.
Revised: Loss for model specialty learning.

Original: The model dispatcher generates ensemble weights to aggregate multiple model predictions for each sample, where models vary in their performance significantly over samples as indicated in Section 2.
Revised: The model dispatcher generates ensemble weights to aggregate multiple model predictions for each sample, where models vary in their performance significantly over samples as indicated in Section 2.

Original: Thus, we expect to assign more weights to the models with higher sample-level specialty to achieve the best utilization of the pretrained models.
Revised: Thus, we expect to assign more weights to the models with higher sample-level specialty to achieve the best utilization of the pretrained models.

Original: We use the likelihood of the ground truth label p ( y i | x i ; θ ′ k ) on the i -th sample produced by model f k as theevaluation metric of its sample-level model specialty .
Revised: We use the likelihood of the ground truth label p ( y i | x i ; θ ′ k ) on the i -th sample produced by model f k as the

Original: [-That is, we try to minimize the estimation risk of the estimated model specialty on the ground truth, i.e., w k and p ( y i | x i ; θ ′ k ) , as L specialty ( ρ ) = − (cid:80) Kk =
 [ p ( y i | x i-]
Revised: [-That is, we try to minimize the estimation risk of the estimated model specialty on the ground truth, i.e., w k and p ( y i | x i ; θ ′ k ) , as L specialty ( ρ ) = − (cid:80) Kk =
 [ p ( y i | x i-][Table]

Original: [Equation]
Revised: [Table]

Original: θ ′ k ) · ln( w k ) + (1 − p ( y i | x i
Revised: evaluation metric of its sample-level model specialty .

Original: [Equation] θ ′ k )) · ln(
Revised: That is, we try to minimize the estimation risk of the estimated model specialty on the ground truth, i.e., w k and p ( y i | x i ; θ ′ k ) , as

Original: [Equation]
Revised: [Equation]

Original: w k )] . [Table] [Table] L specialty is used to optimize the model-sample network and the ensemble layer to work jointly as a specialty-aware model dispatcher.
Revised: L specialty is used to optimize the model-sample network and the ensemble layer to work jointly as a specialty-aware model dispatcher.

Original: Therefore, the total loss to minimize is L = a e L ens ( ψ, ρ )+ a d L adapter ( ψ )+ a s L specialty ( ρ ) , where a e , a d , and a s are loss weights.
Revised: Therefore, the total loss to minimize is L = a e L ens ( ψ, ρ )+ a d L adapter ( ψ )+ a s L specialty ( ρ ) , where a e , a d , and a s are loss weights.

Original: It is worth noting that the only parameters to update are { ψ, ρ } , each of which is lightweight compared to the pretrained models which remain fixed in our method, yet have been fine-tuned in other previous works.
Revised: It is worth noting that the only parameters to update are { ψ, ρ } , each of which is lightweight compared to the pretrained models which remain fixed in our method, yet have been fine-tuned in other previous works.

Original:  E XPERIMENTS 4. E VALUATION P ROTOCOL
Revised:  E XPERIMENTS 4. E VALUATION P ROTOCOL

Original: We conduct experiments on DomainBed suite (Gulrajani & Lopez-Paz, 2020) which provides likefor-like comparisons between algorithms with a standard evaluation, as detailed in Appendix A.6.
Revised: We conduct experiments on DomainBed suite (Gulrajani & Lopez-Paz, 2020) which provides likefor-like comparisons between algorithms with a standard evaluation, as detailed in Appendix A.6.

Original: Datasets.
Revised: Datasets.

Original: We experiment on 5 real-world benchmark datasets including PACS (4 domains, 9, samples, 7 classes)
Revised: We experiment on 5 real-world benchmark datasets including PACS (4 domains, 9, samples, 7 classes)

Original: (Li et al., 2017), VLCS (4 domains, 10,729 samples, 5 classes) (Fang et al., 2013), OfficeHome (4 domains, 15,588 samples, 65 classes) (Venkateswara et al., 2017), TerraIncognita (4 domains, 24,778 samples, 10 classes) (Beery et al., 2018), and DomainNet (6 domains, 586,575 samples, 345 classes) (Peng et al., 2019).
Revised: (Li et al., 2017), VLCS (4 domains, 10,729 samples, 5 classes) (Fang et al., 2013), OfficeHome (4 domains, 15,588 samples, 65 classes) (Venkateswara et al., 2017), TerraIncognita (4 domains, 24,778 samples, 10 classes) (Beery et al., 2018), and DomainNet (6 domains, 586,575 samples, 345 classes) (Peng et al., 2019).

Original: Baselines.
Revised: Baselines.

Original: We compare SIMPLE with strong DG baselines including state-of-the-art.
Revised: We compare SIMPLE with strong DG baselines including state-of-the-art.

Original: General DG methods incorporate elaborate learning algorithms including ERM (Vapnick, 1998), CORAL (Sun & Saenko, 2016), MLDG (Li et al., 2018a), MMD (Li et al., 2018b), DANN (Ganin et al., 2016),
Revised: General DG methods incorporate elaborate learning algorithms including ERM (Vapnick, 1998), CORAL (Sun & Saenko, 2016), MLDG (Li et al., 2018a), MMD (Li et al., 2018b), DANN (Ganin et al., 2016),

Original: C-DANN (Li et al., 2018c), Fish (Shi et al., 2021), LP-FT (Kumar et al., 2021), and MIRO (Cha et al., 2022).
Revised: C-DANN (Li et al., 2018c), Fish (Shi et al., 2021), LP-FT (Kumar et al., 2021), and MIRO (Cha et al., 2022).

Original: Some other works incorporate ensemble learning, including SWAD (Cha et al., 2021) and EoA (Arpit et al., 2021).
Revised: Some other works incorporate ensemble learning, including SWAD (Cha et al., 2021) and EoA (Arpit et al., 2021).

Original: MIRO has combined SWAD (MIRO + SWAD) into their approach, with the result being the current SOTA for DomainBed.
Revised: MIRO has combined SWAD (MIRO + SWAD) into their approach, with the result being the current SOTA for DomainBed.

Original: More details can refer to Appendix A.7. 
Revised: More details can refer to Appendix A.7. [Table]

Original: Model pool composition.
Revised: Model pool composition.

Original: We collect 283 pretrained models to compose model pools, described in detail in Appendix A.3.
Revised: We collect 283 pretrained models to compose model pools, described in detail in Appendix A.3.

Original: Based on their pretraining domains, we divide them into a pure ImageNetpretrained model pool (Model Pool-A) and the one with pretrained models from different datasets 
Revised: Based on their pretraining domains, we divide them into a pure ImageNetpretrained model pool (Model Pool-A) and the one with pretrained models from different datasets (Model Pool-B).

Original: [Table]

Original: (Model Pool-B). We denote SIMPLE using Model Pool-B as SIMPLE + to distinguish it from the one using Model Pool-A.
Revised: We denote SIMPLE using Model Pool-B as SIMPLE + to distinguish it from the one using Model Pool-A.

Original:  E VALUATION RESULTS ON D OMAIN B ED
Revised: 4. E VALUATION RESULTS ON D OMAIN B ED

Original: This section presents the evaluation results of DomainBed suite, on which we compare SIMPLE with general DG algorithms to verify its effectiveness and efficiency.
Revised: This section presents the evaluation results of DomainBed suite, on which we compare SIMPLE with general DG algorithms to verify its effectiveness and efficiency.

Original: Specifically, we compare algorithms using ImageNet-pretrained models only (e.g., SIMPLE using Model Pool-A) and using pretrained models from diverse pretraining datasets (e.g., SIMPLE + using Model Pool-B), resp.
Revised: Specifically, we compare algorithms using ImageNet-pretrained models only (e.g., SIMPLE using Model Pool-A) and using pretrained models from diverse pretraining datasets (e.g., SIMPLE + using Model Pool-B), resp.

Original: Performance comparison.
Revised: Performance comparison.

Original: As shown in Table 1, SIMPLE achieves an average performance of 72.0% with ImageNet-pretrained models, exceeding the current SOTA competitor (MIRO + SWAD) by 3.9% and ranks first in all datasets.
Revised: As shown in Table 1, SIMPLE achieves an average performance of 72.0% with ImageNet-pretrained models, exceeding the current SOTA competitor (MIRO + SWAD) by 3.9% and ranks first in all datasets.

Original: SIMPLE even outperforms the second-best method by 12.2% on OfficeHome dataset.
Revised: SIMPLE even outperforms the second-best method by 12.2% on OfficeHome dataset.

Original: The results show that our proposed paradigm of using only pretrained models without fine-tuning is more effective than the traditional paradigm that requires fine-tuning.
Revised: The results show that our proposed paradigm of using only pretrained models without fine-tuning is more effective than the traditional paradigm that requires fine-tuning.

Original: Furthermore, by using Model Pool-B that includes models pretrained on other datasets, SIMPLE + further improves the average performance by 6.1% over SIMPLE and beats other strong baselines.
Revised: Furthermore, by using Model Pool-B that includes models pretrained on other datasets, SIMPLE + further improves the average performance by 6.1% over SIMPLE and beats other strong baselines.

Original: Training cost comparison.
Revised: Training cost comparison.

Original: As SIMPLE does not fine-tune these models but use the matching network to dispatch models, the training cost is almost negligible versus general DG methods.
Revised: As SIMPLE does not fine-tune these models but use the matching network to dispatch models, the training cost is almost negligible versus general DG methods.

Original: As shown in Table 2, the number of learnable parameters of SIMPLE (0.9M) is minor compared with the normal image backbone network used by MIRO (25.6M for ResNet-50 and 79.7M for RegNetY16GF).
Revised: As shown in Table 2, the number of learnable parameters of SIMPLE (0.9M) is minor compared with the normal image backbone network used by MIRO (25.6M for ResNet-50 and 79.7M for RegNetY16GF).

Original: In addition, training of SIMPLE uses noticeably less time with 1000 times speedup.
Revised: In addition, training of SIMPLE uses noticeably less time with 1000 times speedup.

Original: Inference cost comparison.
Revised: Inference cost comparison.

Original: Additionally the inference cost is controlled through selectively activating appropriate models with the highest ˜ k ( < K ) matching metric values, for inference.
Revised: Additionally the inference cost is controlled through selectively activating appropriate models with the highest ˜ k ( < K ) matching metric values, for inference.

Original: This reduces the inference cost to a large extent, though more than 200 pretrained models have been incorporated in the model pool.
Revised: This reduces the inference cost to a large extent, though more than 200 pretrained models have been incorporated in the model pool.

Original: As shown in Table 2, although SIMPLE uses more inference cost than the model using ResNet-50 as the backbone, SIMPLE + uses less inference time than the existing SOTA (MIRO + SWAD) using RegNetY-16GF (Singh et al., 2022) as the backbone.
Revised: As shown in Table 2, although SIMPLE uses more inference cost than the model using ResNet-50 as the backbone, SIMPLE + uses less inference time than the existing SOTA (MIRO + SWAD) using RegNetY-16GF (Singh et al., 2022) as the backbone.

Original: That is,
Revised: That is,

Original: SIMPLE + obtains new SOTA results with significantly higher training and inference efficiency .
Revised: SIMPLE + obtains new SOTA results with significantly higher training and inference efficiency .

Original: The significant training time advantage of SIMPLE and its surpassing performance indicate that
Revised: The significant training time advantage of SIMPLE and its surpassing performance indicate that

Original: SIMPLE is an effective and efficient paradigm for DG.
Revised: SIMPLE is an effective and efficient paradigm for DG.

Original: We illustrate the matching preference of
Revised: We illustrate the matching preference of

Original: SIMPLE, i.e., which models are typically dispatched in specific domains, in Appendix A.10.
Revised: SIMPLE, i.e., which models are typically dispatched in specific domains, in Appendix A.11.

Original: 4. A BLATION STUDY
Revised: 4. A BLATION STUDY

Original: We verify the effectiveness of SIMPLE design by answering two research questions (RQ) as below.
Revised: We verify the effectiveness of SIMPLE design by answering two research questions (RQ) as below.

Original: (RQ1) Does SIMPLE achieve better generalization performance than the best single model in the model pool?
Revised: (RQ1) Does SIMPLE achieve better generalization performance than the best single model in the model pool?

Original: To verify, we train an individual adapter on source domains for each pretrained model and show the “cheating” upper bound of single model performance by reporting the best single model performance on test set as test-best single model in Table 3.
Revised: To verify, we train an individual adapter on source domains for each pretrained model and show the “cheating” upper bound of single model performance by reporting the best single model performance on test set as test-best single model in Table 3.

Original: From the results, the best single model has outperformed the current SOTA (MIRO + SWAD) on certain datasets (e.g.,
Revised: As shown in Table 1, the best single model has outperformed the current SOTA (MIRO + SWAD) on certain datasets (e.g.,

Original: OfficeHome and VLCS) shown in Table 1.
Revised: OfficeHome and VLCS).

Original: Nevertheless, SIMPLE beats it in all the comparisons ( RQ1 ).
Revised: Nevertheless, SIMPLE beats it in all the comparisons ( RQ1 ).

Original: This implies that SIMPLE has somehow compensated for the drawbacks brought by NFL of a single model in DG, by utilizing different pretrained models and dispatching them selectively to OOD samples.
Revised: This implies that SIMPLE has somehow compensated for the drawbacks brought by NFL of a single model in DG, by utilizing different pretrained models and dispatching them selectively to OOD samples.

Original: (RQ2) Is it necessary to assemble pretrained models according to their specialty?
Revised: (RQ2) Is it necessary to assemble pretrained models according to their specialty?

Original: The analysis in Section 2.1 shows that pretrained models possess certain generalization ability.
Revised: The analysis in Section 2.1 shows that pretrained models possess certain generalization ability.

Original: Thus, a natural approach is to simply aggregate pretrained models without considering their relative specialty on different samples, e.g., randomly select ˜ k models for each sample and average their outputs as the final prediction (Lakshminarayanan et al., 2017).
Revised: Thus, a natural approach is to simply aggregate pretrained models without considering their relative specialty on different samples, e.g., randomly select ˜ k models for each sample and average their outputs as the final prediction (Lakshminarayanan et al., 2017).

Original: The results of the random ensemble are shown in
Revised: The results of the random ensemble are shown in

Original: Table 3, illustrating that random ensemble lags behind SIMPLE by a large margin.
Revised: Table 3, illustrating that random ensemble lags behind SIMPLE by a large margin.

Original: This verifies the necessity to select and ensemble pretrained models based on their specialty over samples ( RQ2 ).
Revised: This verifies the necessity to select and ensemble pretrained models based on their specialty over samples ( RQ2 ).

Original: 4. P RACTICAL TIPS FOR CONSTRUCTING MODEL POOLS
Revised: 4. P RACTICAL TIPS FOR CONSTRUCTING MODEL POOLS

Original: In this section, we investigate the impact of different model pool properties on the generalization performance, to provide useful guidance for the composition and utilization of model pools.
Revised: In this section, we investigate the impact of different model pool properties on the generalization performance, to provide useful guidance for the composition and utilization of model pools.

Original: The corresponding results are presented in Table 3.
Revised: The corresponding results are presented in Table 3.

Original: Specifically, we focus on the size and diversity of model pools by comparing four different model pools, i.e., Model Pool-A-Small (ImageNetpretrained, 15 models), Model Pool-A (ImageNet-pretrained, 244 models), Model Pool-B-Small (diverse pretraining datasets, 17 models), and Model Pool-B (diverse, 283 models).
Revised: Specifically, we focus on the size and diversity of model pools by comparing four different model pools, i.e., Model Pool-A-Small (ImageNetpretrained, 15 models), Model Pool-A (ImageNet-pretrained, 244 models), Model Pool-B-Small (diverse pretraining datasets, 17 models), and Model Pool-B (diverse, 283 models).

Original: The composition of these model pools can be found in Appendix A.3.
Revised: The composition of these model pools can be found in Appendix A.3.

Original: Tip 1: Use a larger model pool.
Revised: Tip 1: Use a larger model pool.

Original: Based on the analysis in Section 2, a larger model pool is favored as it increases the probability that the model pool contains models that match well with each OOD sample.
Revised: Based on the analysis in Section 2, a larger model pool is favored as it increases the probability that the model pool contains models that match well with each OOD sample.

Original: This is consistent with the comparison of Model Pool-A-Small with Model Pool-A, and Model Pool-B-Small with
Revised: This is consistent with the comparison of Model Pool-A-Small with Model Pool-A, and Model Pool-B-Small with

Original: Model Pool-B.
Revised: Model Pool-B.

Original: Both types of model pools show significantly better generalization performance as the size of the model pool increases.
Revised: Both types of model pools show significantly better generalization performance as the size of the model pool increases.

Original: Tip 2: Incorporate diverse models.
Revised: Tip 2: Incorporate diverse models.

Original: The performance comparison of Model Pool-A(-Small and Model Pool-B(-Small indicates that adopting diverse pretrained models 
Revised: The performance comparison of Model Pool-A(-Small) and Model Pool-B(-Small) indicates that adopting diverse pretrained models from different pretraining

Original: from different

Original: pretraining domains can enhance DG performance remarkably.
Revised: domains can enhance DG performance remarkably.

Original: Tip 3: Increasing model diversity matters more than increasing model pool size.
Revised: Tip 3: Increasing model diversity matters more than increasing model pool size.

Original: On top of Model Pool-A-Small, Model Pool- A incorporates 224 additional pretrained models from ImageNet pretraining domain, while Model Pool-B-Small adds 2 more models that are pretrained on
Revised: On top of Model Pool-A-Small, Model Pool-A incorporates 224 additional pretrained models from ImageNet pretraining domain, while Model Pool-B-Small adds 2 more models that are pretrained on

Original: YFCC100M
Revised: YFCC100M

Original: (Radford et al., 2021).
Revised: (Radford et al., 2021).

Original: However, SIMPLE performs better upon Model Pool-B-Small, suggesting that diversity of model pool may be more important for generalization.
Revised: However, SIMPLE performs better upon Model Pool-B-Small, suggesting that diversity of model pool may be more important for generalization.

Original: The above tips from empirical observations that, more models and more diversity can improve generalization performance, again suggest that there is no free lunch for DG.
Revised: The above tips from empirical observations that, more models and more diversity can improve generalization performance, again suggest that there is no free lunch for DG.

Original: Thus, different pretrained models need to be incorporated to address DG.
Revised: Thus, different pretrained models are needed to address DG.

Original: SIMPLE provides a way to realize this goal both effectively and efficiently by (1) adapting pretrained models to unseen domains via label space adaption with a low cost and (2) dispatching best-fit models from a large model pool to handle each OOD sample.
Revised: SIMPLE provides a way to realize this goal both effectively and efficiently by (1) adapting pretrained models to unseen domains via label space adaption with a low cost and (2) dispatching best-fit models from a large model pool to handle each OOD sample.

Original: 4. S ENSITIVITY ANALYSIS
Revised: 4. S ENSITIVITY ANALYSIS

Original: We conduct sensitivity analysis for ˜ k and sample feature extractor f k 0 , detailed in Appendix A.8, and the main findings include: (1) There is a marginal effect on the generalization performance improvement brought by increasing ˜ k , and SIMPLE can outperform SOTA baseline (MIRO + SWAD) even when ˜ k
Revised: We conduct sensitivity analysis for ˜ k and sample feature extractor f k 0 , detailed in Appendix A.9, and the main findings include: (1) There is a marginal effect on the generalization performance improvement brought by increasing ˜ k , and SIMPLE can outperform SOTA baseline (MIRO + SWAD) even when ˜ k

Original: = 2 ; and (2) SIMPLE is robust to the selection of feature extractors.
Revised: = 2 ; and (2) SIMPLE is robust to the selection of feature extractors.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Despite recent studies suggesting that network architectures and pretraining practices affect generalization ability to a large extent, no work has explored the use of these easy-to-obtain pretrained models to address domain generalization.
Revised: Despite recent studies suggesting that network architectures and pretraining practices affect generalization ability to a large extent, no work has explored the use of these easy-to-obtain pretrained models to address domain generalization.

Original: Our work provides a comprehensive analysis of generalizing pretrained models to unseen domains and reveals that there is no free lunch of pretrained models in DG.
Revised: Our work provides a comprehensive analysis of generalizing pretrained models to unseen domains and reveals that there is no free lunch of pretrained models in DG.

Original: Based on that, we propose a novel DG paradigm that leverages fixed pretrained models and dispatches them to OOD samples based on their matching metric to the target task.
Revised: Based on that, we propose a novel DG paradigm that leverages fixed pretrained models and dispatches them to OOD samples based on their matching metric to the target task.

Original: Extensive evaluations show that our proposal is a promising alternative for DG, with better generalization performance and significantly higher training efficiency compared to existing DG methods.
Revised: Extensive evaluations show that our proposal is a promising alternative for DG, with better generalization performance and significantly higher training efficiency compared to existing DG methods.

Original: A A PPENDIX A. R ELATED WORK Mainstream
Revised: A A PPENDIX A. R ELATED WORK Mainstream

Original: DG research generally addresses distributional shifts through data manipulation (Tobin et al., 2017; Peng et al., 2018), robust representational learning (Tzeng et al., 2014; Arjovsky et al., 2019) and exploration of new learning strategies such as ensemble learning Cha et al.
Revised: DG research generally addresses distributional shifts through data manipulation (Tobin et al., 2017; Peng et al., 2018), robust representational learning (Tzeng et al., 2014; Arjovsky et al., 2019) and exploration of new learning strategies such as ensemble learning Cha et al.

Original: We here only review studies that are mainstream in DG and that relate to our method, for a more comprehensive survey refer to Wang et al.
Revised: We here only review studies that are mainstream in DG and that relate to our method, for a more comprehensive survey refer to Wang et al.

Original: Data manipulation.
Revised: Data manipulation.

Original: DG arises from the lack of sufficient data to feed machine learning models, so they are biased to the training distribution, resulting in an inability to work well on the test distribution (Wang et al., 2022).
Revised: DG arises from the lack of sufficient data to feed machine learning models, so they are biased to the training distribution, resulting in an inability to work well on the test distribution (Wang et al., 2022).

Original: Thus, various methods resort to manipulating the training samples to simulate unseen target domains, among which there are two techniques, namely data augmentation (Shankar et al., 2018) and data generation (Li et al., 2021).
Revised: Thus, various methods resort to manipulating the training samples to simulate unseen target domains, among which there are two techniques, namely data augmentation (Shankar et al., 2018) and data generation (Li et al., 2021).

Original: Data augmentation has been widely used in general machine learning models to enhance their generalization ability by avoiding overfitting (Shorten & Khoshgoftaar, 2019).
Revised: Data augmentation has been widely used in general machine learning models to enhance their generalization ability by avoiding overfitting (Shorten & Khoshgoftaar, 2019).

Original: Inspired by its importance, Tobin et al.
Revised: Inspired by its importance, Tobin et al.

Original: (2017) first uses it for DG to simulate test distributions, and subsequently, Peng et al.
Revised: (2017) first uses it for DG to simulate test distributions, and subsequently, Peng et al.

Original: Khirodkar et al.
Revised: Khirodkar et al.

Original: Tremblay et al.
Revised: Tremblay et al.

Original: (2018) also adopt data augmentation in various ways.
Revised: (2018) also adopt data augmentation in various ways.

Original: In addition to the common feature augmentation, Peng et al.
Revised: In addition to the common feature augmentation, Peng et al.

Original: (2022) further propose to perform label augmentation.
Revised: (2022) further propose to perform label augmentation.

Original: In the field of data generation, instead of augmentation, methods generate new samples or domains by means of such as Mixup (Zhang et al., 2017), auto-encoder (Qiao et al., 2020), generative adversarial network (Rahman et al., 2019).
Revised: In the field of data generation, instead of augmentation, methods generate new samples or domains by means of such as Mixup (Zhang et al., 2017), auto-encoder (Qiao et al., 2020), generative adversarial network (Rahman et al., 2019).

Original: Representation learning.
Revised: Representation learning.

Original: In another promising direction, some DG studies attempt to extract domain-invariant features that can be generalized to unseen domains (Ben-David et al., 2006), or separate domain-shared and domain-specific parts from the features for generalization (Khosla et al., 2012).
Revised: In another promising direction, some DG studies attempt to extract domain-invariant features that can be generalized to unseen domains (Ben-David et al., 2006), or separate domain-shared and domain-specific parts from the features for generalization (Khosla et al., 2012).

Original: Motivated by the transferability of domain-invariant features across varied domains and the expectation of their generalization ability to unseen domains, methods try to obtain a specific feature space that is invariant to domain labels.
Revised: Motivated by the transferability of domain-invariant features across varied domains and the expectation of their generalization ability to unseen domains, methods try to obtain a specific feature space that is invariant to domain labels.

Original: Li et al.
Revised: Li et al.

Original: (2018b) first introduces adversarial training in DG, allowing the generator seeks to trick the discriminator about the domain labels of images and thus generate domain-invariant features.
Revised: (2018b) first introduces adversarial training in DG, allowing the generator seeks to trick the discriminator about the domain labels of images and thus generate domain-invariant features.

Original: Several works (Shao et al., 2019; Rahman et al., 2020; Wang et al., 2020b) follow this direction.
Revised: Several works (Shao et al., 2019; Rahman et al., 2020; Wang et al., 2020b) follow this direction.

Original: In the line of extracting domain-invariant features, other works explicitly align feature distribution learned from source domains, with the differences measured by
Revised: In the line of extracting domain-invariant features, other works explicitly align feature distribution learned from source domains, with the differences measured by

Original: Wasserstein distance (Zhou et al., 2020), maximum mean discrepancy (Wang et al., 2018; 2020a), or mean and variance (Peng et al., 2019). 
Revised: Wasserstein distance (Zhou et al., 2020), maximum mean discrepancy (Wang et al., 2018; 2020a), or mean and variance (Peng et al., 2019). At a meta level, Balaji et al.

Revised: (2018) consider learning a regularization function on classifier to avoid biasing to domain-specific information.

Original: In contrast, for robust representation learning, some work aims to separate domain-invariant variables from the specific ones of the learned features (Niu et al., 2015; Ilse et al., 2020;
Revised: In contrast, for robust representation learning, some work aims to separate domain-invariant variables from the specific ones of the learned features (Niu et al., 2015; Ilse et al., 2020;

Original: He et al., 2021b).
Revised: He et al., 2021b).

Original: Ensemble learning.
Revised: Ensemble learning.

Original: Ensemble learning methods Hansen & Salamon (1990)
Revised: Ensemble learning methods Hansen & Salamon (1990)

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2018) exploit multiple models to produce prediction results and combine the results with various techniques, e.g., boosting Schapire (1990); Freund (1995); Moghimi et al.
Revised: (2018) exploit multiple models to produce prediction results and combine the results with various techniques, e.g., boosting Schapire (1990); Freund (1995); Moghimi et al.

Original: (2016) or
Revised: (2016) or

Original: mean aggregation Zhou et al.
Revised: mean aggregation Zhou et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2020), aiming at achieving better performance than individual model alone.
Revised: (2020), aiming at achieving better performance than individual model alone.

Original: As a well-known technique to improve generalization performance, ensemble learning has also been explored in DG.
Revised: As a well-known technique to improve generalization performance, ensemble learning has also been explored in DG.

Original: One general idea of using ensemble learning by existing DG methods is to utilize the relationship between unseen domains and source domains.
Revised: One general idea of using ensemble learning by existing DG methods is to utilize the relationship between unseen domains and source domains.

Original: Specifically, in the work of Mancini et al.
Revised: Specifically, in the work of Mancini et al.

Original: (2018), they train an individual classifier for each specific source domain and one additional classifier to predict the probability of each domain that a sample belongs to, and aggregate their predictions by weights accordingly.
Revised: (2018), they train an individual classifier for each specific source domain and one additional classifier to predict the probability of each domain that a sample belongs to, and aggregate their predictions by weights accordingly.

Original: In addition, Segu et al.
Revised: In addition, Segu et al.

Original: (2020) proposes to maintain domain-specific batch normalization layers, which will be weighted for aggregation in inference.
Revised: (2020) proposes to maintain domain-specific batch normalization layers, which will be weighted for aggregation in inference.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2021) instead trains only domain-specific classifier heads while letting them share the same feature extractor.
Revised: (2021) instead trains only domain-specific classifier heads while letting them share the same feature extractor.

Original: SWAD (Cha et al., 2021) and EoA (Arpit et al., 2021) instead use model ensemble and weight ensemble directly to improve the generalization.
Revised: SWAD (Cha et al., 2021) and EoA (Arpit et al., 2021) instead use model ensemble and weight ensemble directly to improve the generalization.

Original: Though promising in enhancing performance, ensemble learning is criticized for its computational efficiency, which hinders the practical application (Wang et al., 2022).
Revised: Though promising in enhancing performance, ensemble learning is criticized for its computational efficiency, which hinders the practical application (Wang et al., 2022).

Original: Relation to existing work.
Revised: Relation to existing work.

Original: The main difference between the existing DG works and SIMPLE resides in the fact that these efforts seek to find an optimal model that can settle all distributional shifts.
Revised: The main difference between the existing DG works and SIMPLE resides in the fact that these efforts seek to find an optimal model that can settle all distributional shifts.

Original: Even DG algorithms that use ensemble learning are trying to find an optimal ensemble that has a flatter loss landscape and better generalization ability (Cha et al., 2021).
Revised: Even DG algorithms that use ensemble learning are trying to find an optimal ensemble that has a flatter loss landscape and better generalization ability (Cha et al., 2021).

Original: To this end, existing works use a specific pretrained model for initialization, as this performs better than training from scratch (Wortsman et al., 2022), and then fine-tune it with training algorithms like data manipulation, robust representation learning, or ensemble learning as discussed above.
Revised: To this end, existing works use a specific pretrained model for initialization, as this performs better than training from scratch (Wortsman et al., 2022), and then fine-tune it with training algorithms like data manipulation, robust representation learning, or ensemble learning as discussed above.

Original: In contrast, recent studies have shown that such specific pretrained models may not be sufficient to solve distributional shifts, and that pretraining strategies need to be chosen for different types of distributional shifts.
Revised: In contrast, recent studies have shown that such specific pretrained models may not be sufficient to solve distributional shifts, and that pretraining strategies need to be chosen for different types of distributional shifts.

Original: To the best of our knowledge, SIMPLE is the first work to address DG by directly using different pretrained models without fine-tuning, by reformulating DG as a model-sample matching problem.
Revised: To the best of our knowledge, SIMPLE is the first work to address DG by directly using different pretrained models without fine-tuning, by reformulating DG as a model-sample matching problem.

Original: Moreover,
Revised: Moreover,

Original: SIMPLE does not fine-tune these pretrained models, but uses domain-level label space adaptation to achieve effective adaptation, which has significantly reduced the cost of training and adaptation.
Revised: SIMPLE does not fine-tune these pretrained models, but uses domain-level label space adaptation to achieve effective adaptation, which has significantly reduced the cost of training and adaptation.

Original: A. M ORE DISCUSSION ABOUT SIMPLE
Revised: A. M ORE DISCUSSION ABOUT SIMPLE

Original: Towards a robustness algorithm that can effectively and efficiently leverage extensive pretrained models for DG, there are certain challanges that remain in our paper.
Revised: Towards a robustness algorithm that can effectively and efficiently leverage extensive pretrained models for DG, there are certain challanges that remain in our paper.

Original: Currently, SIMPLE is not feasible for adding new pretrained models into model pool for direct utilization.
Revised: Currently, SIMPLE is not feasible for adding new pretrained models into model pool for direct utilization.

Original: Although retraining is lightweight, we need a more straightforward approach to address this problem due to the rapid development of pre-trained models.
Revised: Although retraining is lightweight, we need a more straightforward approach to address this problem due to the rapid development of pre-trained models.

Original: We leave this problem, i.e., cold-start problem which has been long studied in recommendation fields, as further work.
Revised: We leave this problem, i.e., cold-start problem which has been long studied in recommendation fields, as further work.

Original: A. P RETRAINED MODEL POOLS
Revised: A. P RETRAINED MODEL POOLS

Original: This section presents the pretrained models used in SIMPLE.
Revised: This section presents the pretrained models used in SIMPLE.

Original: With more and more pretrained models being published, it is straightforward to build a pretrained model pool consisting of several public pretrained models for direct adaptation to novel domains.
Revised: With more and more pretrained models being published, it is straightforward to build a pretrained model pool consisting of several public pretrained models for direct adaptation to novel domains.

Original: In particular, these models are categorized according to their pretraining domain as an ImageNet-pretrained model pool (Model Pool-A) and a pool (Model Pool-B) containing models pretrained on datasets other than the ImageNet dataset.
Revised: In particular, these models are categorized according to their pretraining domain as an ImageNet-pretrained model pool (Model Pool-A) and a pool (Model Pool-B) containing models pretrained on datasets other than the ImageNet dataset.

Original: Model Pool-A. DG methods commonly use ImageNet-pretrained models to initialize the model weights for further fine-tuning(Kim et al., 2021).
Revised: Model Pool-A. DG methods commonly use ImageNet-pretrained models to initialize the model weights for further fine-tuning(Kim et al., 2021).

Original: To present fair comparisons with those methods, we first construct an ImageNet-pretrained model pool (Model Pool-A).
Revised: To present fair comparisons with those methods, we first construct an ImageNet-pretrained model pool (Model Pool-A).

Original: Specifically, we collectmodels with diverse network architectures and learning objectives from popular 3rd-party repositories 1 23 .
Revised: Specifically, we collectmodels with diverse network architectures and learning objectives from popular 3rd-party repositories 3 45 .

Original: Note that, the models pretrained on the same dataset, i.e., ImageNet, will share the same label adapter transforming the vanilla model outputs to the target label space.
Revised: Note that, the models pretrained on the same dataset, i.e., ImageNet, will share the same label adapter transforming the vanilla model outputs to the target label space.

Original: Therefore, for Model
Revised: Therefore, for Model

Original: Pool-A, only one label space adapter will be trained, to transform the original predictions for 1, classes in ImageNet into the predictions of the target class space.
Revised: Pool-A, only one label space adapter will be trained, to transform the original predictions for 1, classes in ImageNet into the predictions of the target class space.

Original: Model Pool-B. Previous studies also found that ImageNet-pretrained model (e.g, ResNet-50) is not sufficient for generalization (Kumar et al., 2021; Kim et al., 2022), suggesting the need for models pretrained on different datasets.
Revised: Model Pool-B. Previous studies also found that ImageNet-pretrained model (e.g, ResNet-50) is not sufficient for generalization (Kumar et al., 2021; Kim et al., 2022), suggesting the need for models pretrained on different datasets.

Original: Therefore, we further expand the Model Pool-A by incorporating models that trained on datasets including ImageNet-21k,
Revised: Therefore, we further expand the Model Pool-A by incorporating models that trained on datasets including ImageNet-21k,

Original: YFCC100M
Revised: YFCC100M

Original: (Radford et al., 2021), Instagram 3.6B (Singh et al., 2022), as Model Pool-B.
Revised: (Radford et al., 2021), Instagram 3.6B (Singh et al., 2022), as Model Pool-B.

Original: In total, we add 39 models, whose weights are obtained from the same sources or from their public repositories 45 .
Revised: In total, we add 39 models, whose weights are obtained from the same sources or from their public repositories 67 .

Original: Thus, Model Pool-B contains 283 pretrained models with varied network architectures, learning objectives, and pretraining datasets.
Revised: Thus, Model Pool-B contains 283 pretrained models with varied network architectures, learning objectives, and pretraining datasets.

Original: To analyze the impact of model pool size on performance, we take a portion of the models from these two model pools and use them to construct Model Pool-A-Small and Model Pool-B-Small.
Revised: To analyze the impact of model pool size on performance, we take a portion of the models from these two model pools and use them to construct Model Pool-A-Small and Model Pool-B-Small.

Original: Specificially, in Model Pool-A-Small , we incorporate the architectures including AlexNet (1) (Krizhevsky et al., 2012), DenseNet-121/169/201 (3) (Iandola et al., 2014), Dual-Path-Network-68 (1) (Chen et al., 2017), NASNetMobile (1) (Zoph et al., 2018), ResNet-18/34/50 (3)
Revised: Specificially, in Model Pool-A-Small , we incorporate the architectures including AlexNet (1) (Krizhevsky et al., 2012), DenseNet-121/169/201 (3) (Iandola et al., 2014), Dual-Path-Network-68 (1) (Chen et al., 2017), NASNetMobile (1) (Zoph et al., 2018), ResNet-18/34/50 (3)

Original: (He et al., 2016), SEResNet-50 (1) Hu et al.
Revised: (He et al., 2016), SEResNet-50 (1) Hu et al.

Original: (2018), SqueezeNet-1.0/1.1 (2) (Iandola et al., 2016), and MAE-ViTBase/Large/Huge (3) (He et al., 2021a).
Revised: (2018), SqueezeNet-1.0/1.1 (2) (Iandola et al., 2016), and MAE-ViTBase/Large/Huge (3) (He et al., 2021a).

Original: On the other hand, several DG algorithms also use models pretrained on other datasets, such as 1G-1B (Arpit et al., 2021) and ILSVRC12 (Thomas et al., 2021).
Revised: On the other hand, several DG algorithms also use models pretrained on other datasets, such as 1G-1B (Arpit et al., 2021) and ILSVRC12 (Thomas et al., 2021).

Original: Therefore, we build Model Pool-B-Small which contains two more CLIP models (Radford et al., 2021), ViT-B/16 and ViT-B/32, which are trained on a subset of the YFCC100M dataset of roughly the same size as ImageNet.
Revised: Therefore, we build Model Pool-B-Small which contains two more CLIP models (Radford et al., 2021), ViT-B/16 and ViT-B/32, which are trained on a subset of the YFCC100M dataset of roughly the same size as ImageNet.

Original: A detailed list of pretrained models is given in table 7, with the sources of the models, the dimensions of their outputs and FLOPs(floating point operations per second).
Revised: A detailed list of pretrained models is given in table 7, with the sources of the models, the dimensions of their outputs and FLOPs

Original: A. E MPIRICAL EVIDENCES OF NO FREE LUNCH
Revised: A. E MPIRICAL EVIDENCES OF NO FREE LUNCH

Original: In this section, we elaborate on the empirical analysis in Section 2, and provide more experimental details and insights for the analysis.
Revised: In this section, we elaborate on the empirical analysis in Section 2, and provide more experimental details and insights for the analysis.

Original: Transfer and performance measurement of pretrained models.
Revised: Transfer and performance measurement of pretrained models.

Original: As discussed in Section 2, instead of fine-tuning or conducting linear probing to adapt those pretrained models to be a predictive model for unseen target domains, we only train a label space adapter that learns the mapping function h ψ .
Revised: As discussed in Section 2, instead of fine-tuning or conducting linear probing to adapt those pretrained models to be a predictive model for unseen target domains, we only train a label space adapter that learns the mapping function h ψ .

Original: Then we train this shared adapter h
Revised: Then we train this shared adapter h

Original: ψ with the empirical loss ˆ E D s ( ψ, ρ ) without fine-tuning the pretrained model parameters { θ k } , as formally described in Section 3.2.
Revised: ψ with the empirical loss ˆ E D s ( ψ, ρ ) without fine-tuning the pretrained model parameters { θ k } , as formally described in Section 3.2.

Original: With the trained adapter, we use the likelihood of the ground truth label p ( y i | x i ; θ ′ k ) on the i -th sample produced by each adapted model, which also indicates the confidence of the ground truth label
Revised: With the trained adapter, we use the likelihood of the ground truth label p ( y i | x i ; θ ′ k ) on the i -th sample produced by each adapted model, which also indicates the confidence of the ground truth label

Original: y i of the model and (cid:80)
Revised: y i of the model and (cid:80)

Original: y ∈Y p ( y | x i ; θ ′ k )
Revised: y ∈Y p ( y | x i ; θ ′ k )

Original: = 1 .
Revised: = 1 .

Original: We utilize this likelihood as the evaluation metric of its sample-level model specialty .
Revised: We utilize this likelihood as the evaluation metric of its sample-level model specialty .

Original: Experimental settings.
Revised: Experimental settings.

Original: First, we analyze the specialty distribution of each pretrained model from an aggregation view (i.e, domains and classes, respectively), and we verify if there exists a dominant pretrained model that generalizes best across different unseen domains.
Revised: First, we analyze the specialty distribution of each pretrained model from an aggregation view (i.e, domains and classes, respectively), and we verify if there exists a dominant pretrained model that generalizes best across different unseen domains.

Original: We calculate domain-level model specialty as summation of the sample-level specialty over all domain samples as (cid:80) ( x i ,y i ) ∼ D log p ( y i | x i ; θ ′ k ) , on TerraIncognita (Beery et al., 2018) with four domains.
Revised: We calculate domain-level model specialty as summation of the sample-level specialty over all domain samples as (cid:80) ( x i ,y i ) ∼ D log p ( y i | x i ; θ ′ k ) , on TerraIncognita (Beery et al., 2018) with four domains.

Original: To reflect the relative model performance, we perform min-max normalization for model specialty values in the same domain.
Revised: To reflect the relative model performance, we perform min-max normalization for model specialty values in the same domain.

Original: These complete results are shown in Figure 4, with only partial results put in
Revised: These complete results are shown in Figure 4, with only partial results put in

Original: Section 2 for a clearer presentation.
Revised: Section 2 for a clearer presentation.

Original: Then, we further examine whether performance divergence also exists at the finer-grained class-level.
Revised: Then, we further examine whether performance divergence also exists at the finer-grained class-level.

Original: Similar as that at domain-level, Figure 4 presents the relative model performance on 10 classes in TerraIncognita.
Revised: Similar as that at domain-level, Figure 4 presents the relative model performance on 10 classes in TerraIncognita.

Original: In addition, to clearly compare model specialty differences at the two levels, we present heatmaps of specialty differences (measured by
Revised: In addition, to clearly compare model specialty differences at the two levels, we present heatmaps of specialty differences (measured by

Original: Kullback-Leibler divergence) for domain and class pairs, respectively in Figure 5.
Revised: Kullback-Leibler divergence) for domain and class pairs, respectively in Figure 5.

Original: Based on the results of comparing the performance of all pretrained models on different domains and classes, we can draw more empirical insights as follows:
Revised: Based on the results of comparing the performance of all pretrained models on different domains and classes, we can draw more empirical insights as follows:

Original: (1) Most importantly, there is no single model that performs best both across domains and classes; (2) The divergence of the performance distribution is significantly more prominent at the class level than at the domain level, as evidenced by the comparison of the Figure 5(a) and (b); (3) As a side finding, models that perform well in the ‘Bird’ class generally perform well in the ‘Bobcat’ class, as indicated by their pair-wise Kullback-Leibler divergence values.
Revised: (1) Most importantly, there is no single model that performs best both across domains and classes; (2) The divergence of the performance distribution is significantly more prominent at the class level than at the domain level, as evidenced by the comparison of the Figure 5(a) and (b); (3) As a side finding, models that perform well in the ‘Bird’ class generally perform well in the ‘Bobcat’ class, as indicated by their pair-wise Kullback-Leibler divergence values.

Original: However, models that perform well in ‘Bird’ and ‘Bobcat’ usually perform poorly in ‘Dog’ and ‘Rabbit’ classes.
Revised: However, models that perform well in ‘Bird’ and ‘Bobcat’ usually perform poorly in ‘Dog’ and ‘Rabbit’ classes.

Original: This can be viewed as another piece of evidence of NFL for shifting distribution shifts.
Revised: This can be viewed as another piece of evidence of NFL for shifting distribution shifts.

Original: A. N O FREE LUNCH THEOREM
Revised: A. N O FREE LUNCH THEOREM

Original: We here detail the formal version of Theorem 1 with related definitions.
Revised: We here detail the formal version of Theorem 1 with related definitions.

Original: Considering a kernel regression task, training samples { x µ , y µ } are sampled from i.i.d.
Revised: Considering a kernel regression task, training samples { x µ , y µ } are sampled from i.i.d.

Original: and the labels which contain noisy are generated from a target function y µ = ¯ f ( x µ )
Revised: and the labels which contain noisy are generated from a target function y µ = ¯ f ( x µ )

Original: + ϵ µ where the noise covariance is ⟨ ϵ µ ϵ ν ⟩ = ε 2 δ µν .
Revised: + ϵ µ where the noise covariance is ⟨ ϵ µ ϵ ν ⟩ = ε 2 δ µν .

Original: Then the model for regression is trained by minimizing the empirical error (ERM loss):
Revised: Then the model for regression is trained by minimizing the empirical error (ERM loss):

Original: [Equation]
Revised: [Equation]

Original: where H is a Reproducing Kernel Hilbert Space (RKHS) associated with a positive semi-definite kernel K ( x, x ′ )
Revised: where H is a Reproducing Kernel Hilbert Space (RKHS) associated with a positive semi-definite kernel K ( x, x ′ )

Original: R D × R D
Revised: R D × R D

Original: → R , and ⟨ ., . )
Revised: → R , and ⟨ ., . )

Original: H is the Hilbert inner product.
Revised: H is the Hilbert inner product.

Original: The generalization error on the test distribution ˜ p
Revised: The generalization error on the test distribution ˜ p

Original: ( x ) is E g
Revised: ( x ) is E g

Original: ( D ) =
Revised: ( D ) =

Original: (cid:68)(cid:0) f ∗ ( x ) − ¯ f ( x ) (cid:1) 2 (cid:69) ˜
Revised: (cid:68)(cid:0) f ∗ ( x ) − ¯ f ( x ) (cid:1) 2 (cid:69) ˜

Original: p ( x ) , which is a random variable whose value depends on the sampled training dataset.
Revised: p ( x ) , which is a random variable whose value depends on the sampled training dataset.

Original: Therefore, the generalization error is averaged over the distribution of all datasets with sample size P :
Revised: Therefore, the generalization error is averaged over the distribution of all datasets with sample size P :

Original: [Equation]
Revised: [Equation]

Original: Based on the problem setting, Canatar et al.
Revised: Based on the problem setting, Canatar et al.

Original: (2021) proposes the following proposition:
Revised: (2021) proposes the following proposition:

Original: Proposition 1.
Revised: Proposition 1.

Original: (Restating Proposition 1 in Canatar et al.
Revised: (Restating Proposition 1 in Canatar et al.

Original: Consider the kernel regression problem defined above, where the model is trained on p ( x ) and tested on ˜ p
Revised: Consider the kernel regression problem defined above, where the model is trained on p ( x ) and tested on ˜ p

Original: ( x ) .
Revised: ( x ) .

Original: Consider the Mercer decomposition of the RKHS kernel K ( x, x ′ )
Revised: Consider the Mercer decomposition of the RKHS kernel K ( x, x ′ )

Original: =
Revised: =

Original: Φ ( x ) T ΛΦ ( x ′ ) , where we defined
Revised: Φ ( x ) T ΛΦ ( x ′ ) , where we defined

Original: M × M ( M possibly infinite) diagonal eigenvalue matrix Λ ργ = η ρ δ ργ and the column vector
Revised: M × M ( M possibly infinite) diagonal eigenvalue matrix Λ ργ = η ρ δ ργ and the column vector

Original: [Figure]
Revised: [Figure]

Original: Φ ( x ) =
Revised: Φ ( x ) =

Original: ( ϕ 1 ( x ) , ϕ 2 ( x ) . .
Revised: ( ϕ 1 ( x ) , ϕ 2 ( x ) . .

Original: ϕ M ( x )) , with (cid:82) dxp ( x ) Φ ( x ) Φ ( x ) ⊤
Revised: ϕ M ( x )) , with (cid:82) dxp ( x ) Φ ( x ) Φ ( x ) ⊤

Original: =
Revised: =

Original: I .
Revised: I .

Original: Also consider an expansion of the target function in the Mercer basis f ( x ) = a ⊤ Φ ( x )  .
Revised: Also consider an expansion of the target function in the Mercer basis f ( x ) = a ⊤ Φ ( x ) with a coefficient vector a .

Original: The dataset averaged OOD generalization error is given by:
Revised: The dataset averaged OOD generalization error is given by:

Revised: [Equation]

Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: where κ must be solved self-consistently
Revised: where κ must be solved self-consistently

Original: [Equation]
Revised: [Equation]

Original: and we defined the M × M overlap matrix
Revised: and we defined the M × M overlap matrix

Original: [Equation]
Revised: [Equation]

Original: Here E 0 ,p ( x ) g denotes the generalization error when both training and test distributions are matched to p ( x )  and is given by:
Revised: Here E 0 ,p ( x ) g denotes the generalization error when both training and test distributions are matched to p ( x ) (i.e., in-distribution error) and is given by:

Original: [Equation]
Revised: [Equation]

Original: Further, the expected estimator is:
Revised: Further, the expected estimator is:

Original: [Equation]
Revised: [Equation]

Original: A. E VALUATION SETTINGS
Revised: A. E VALUATION SETTINGS

Original: Evaluation protocol.
Revised: Evaluation protocol.

Original: We follow DomainBed protocol to conduct our evaluation, for fair comparisons with baselines.
Revised: We follow DomainBed protocol to conduct our evaluation, for fair comparisons with baselines.

Original: We use the training-domain validation set protocol for model selection.
Revised: We use the training-domain validation set protocol for model selection.

Original: Specifically, one domain in a dataset is selected as the target domain and the rest as source domains, from which 20% of samples are used as the validation set.
Revised: Specifically, one domain in a dataset is selected as the target domain and the rest as source domains, from which 20% of samples are used as the validation set.

Original: All runs are repeated 3 times using different random seeds, thus, with different train-validation splits.
Revised: All runs are repeated 3 times using different random seeds, thus, with different train-validation splits.

Original: The out-of-domain test performance averaged over all domains will be reported for each dataset.
Revised: The out-of-domain test performance averaged over all domains will be reported for each dataset.

Original: In addition, we use the standard number of iterations of 5,000 for all datasets, with early-stop based on validated accuracy.
Revised: In addition, we use the standard number of iterations of 5,000 for all datasets, with early-stop based on validated accuracy.

Original: Hyperparameter tuning.
Revised: Hyperparameter tuning.

Original: Here we state the details of hyperparameter tuning in our experiments.
Revised: Here we state the details of hyperparameter tuning in our experiments.

Original: We use separate Adam optimizers (Zhang, 2018) for the ensemble network and label space adapter.
Revised: We use separate Adam optimizers (Zhang, 2018) for the ensemble network and label space adapter.

Original: Table 4 lists the hyperparameters to tune and their search space.
Revised: Table 4 lists the hyperparameters to tune and their search space.

Original: For each domain, we sweep through 48 different hyperparameter settings. 
Revised: For each domain, we sweep through 48 different hyperparameter settings. [Table]

Original: A. B ASELINES
Revised: A. B ASELINES

Original: • Stochastic Weight Averaging Densely (SWAD) (Cha et al., 2021): SWAD performs weight ensemble during model training.
Revised: • Stochastic Weight Averaging Densely (SWAD) (Cha et al., 2021): SWAD performs weight ensemble during model training.

Original: [Table]

Original: • Ensemble of Average (EoA) (Arpit et al., 2021): EoA combines both model ensemble and weight ensemble by taking an ensemble of moving average models from 6 runs.
Revised: • Ensemble of Average (EoA) (Arpit et al., 2021): EoA combines both model ensemble and weight ensemble by taking an ensemble of moving average models from 6 runs.

Original: They experiment with three different pretrained models as initialization.
Revised: They experiment with three different pretrained models as initialization.

Original: The first one is pretrained on ImageNet with ResNet-50 and the second is pretrained on both ImageNet and a much larger additional dataset, IG-1B, with a more advanced backbone, ResNeXt-50 (Xie et al., 2017).
Revised: The first one is pretrained on ImageNet with ResNet-50 and the second is pretrained on both ImageNet and a much larger additional dataset, IG-1B, with a more advanced backbone, ResNeXt-50 (Xie et al., 2017).

Original: Additionally, with a pretrained RegNetY-16GF (Singh et al., 2022), EoA achieves its best results.
Revised: Additionally, with a pretrained RegNetY-16GF ( [Table] et al., 2022), EoA achieves its best results.

Original: We denote the last one as EoA + to indicate it uses the additional dataset and to compare with SIMPLE + .
Revised: We denote the last one as EoA + to indicate it uses the additional dataset and to compare with SIMPLE + .

Original: • Random ensemble:
Revised: • Random ensemble:

Original: In contrast to SIMPLE of learning to select models for ensemble, we also compare it with an average ensemble of  k models chosen randomly for each sample.
Revised: In contrast to SIMPLE of learning to select models for ensemble, we also compare it with an average ensemble of ˜ k models chosen randomly for each sample.

Revised: A. M ORE ABLATION STUDY

Revised: Here we introduce an additional baseline that uses the same set of ensemble weights for all samples in a domain, rather than generating different ensemble weights to aggregate model predictions differently for each OOD sample.

Revised: For ensemble-based approaches, the overall prediction is given by ˆ y i =

Revised: w ⊤ [ f 1 ( x i ) , · · · , f K ( x i )] , where w k is the weight for aggregating the prediction of k -th models f k ( x i ) .

Original: A. S ENSITIVITY ANALYSIS A.8. A NALYSIS OF ˜ k IN TOP - ˜ k MODEL SELECTION
Revised: In SIMPLE, the ensemble weights are given by w = MLP ( c ⊤ i C ) , where c i is the embedding for the i -th sample and C contains the embeddings for all the models. Therefore, there is a special case of proposed ensemble approach, where the ensemble weights w ∈ R K are randomly initialized and optimized through back-propagation. Here in this simplified version, w is shared for all the samples in the dataset and does not incorporate model or sample information.

Original: Intuitively, using more  models  in an ensemble may lead to better performance (Zhang et al., 2020).
Revised: Compared with this simplified version, SIMPLE explicitly leverages sample and model information (encoded in the embedding vectors c i and C ) to generate specialized ensemble weights for each sample, which is more fine-grained. With model and sample embedding, SIMPLE enjoys much lower training cost when incorporating new pretrained models not in the model pool.

Original: However
Revised: We implement this special case and conduct experiments on OfficeHome dataset to compare with SIMPLE. By performing sufficient hyperparameter tuning, such a simple version achieves an average accuracy of 82.3% on OfficeHome, which is worse than 87.7% of SIMPLE (and even worse than 84.6% of SIMPLE using the much smaller Model Pool-A) and 83.9% of the existing SOTA.

Original: [Table]
Revised: It in contrast suggests that, incorporating information about models and the sample to conduct finegrained model-sample matching in SIMPLE, is necessary and more effective, than using a set of weights and optimized by back-propagation. A. S ENSITIVITY ANALYSIS A.9. A NALYSIS OF ˜ k IN TOP - ˜ k MODEL SELECTION Intuitively, using more models in an ensemble may lead to better performance (Zhang et al., 2020).

Original:  a larger ensemble size also means that the ensemble consumes more computation in inference.
Revised: However, a larger ensemble size also means that the ensemble consumes more computation in infer- ence.

Original: Thus, in inference [Table] we choose to selectively activate
Revised: Thus, in inference, we choose to selectively activate

Original: ˜ k
Revised: ˜ k

Original: ( < K ) models with higher ensemble weights.
Revised: ( < K ) models with higher ensemble weights.

Original: It is necessary to verify the impact of how many models are used for the prediction of each sample on the final performance, which can help balance the effectiveness and efficiency of our approach.
Revised: It is necessary to verify the impact of how many models are used for the prediction of each sample on the final performance, which can help balance the effectiveness and efficiency of our approach.

Original: Settings.
Revised: Settings.

Original: To demonstrate the sensitivity of changes in the parameter ˜ k (the number of models activated in inference) to the final generalization performance, we evaluated the performance of
Revised: To demonstrate the sensitivity of changes in the parameter ˜ k (the number of models activated in inference) to the final generalization performance, we evaluated the performance of

Original: SIMPLE + with different values of ˜ k in the top-  k selection on the OfficeHome dataset.
Revised: SIMPLE + with different values of ˜ k in the top- ˜ k selection on the OfficeHome dataset.

Original: Specifically, we evaluate ˜ k ∈ [1 , . .
Revised: Specifically, we evaluate ˜ k ∈ [1 , . .

Original: , 10] , with a limited and average sweep of the hyperparameters to save computational time.
Revised: , 10] , with a limited and average sweep of the hyperparameters to save computational time.

Original: Therefore, the accuracy may not be optimal.
Revised: Therefore, the accuracy may not be optimal.

Original: # Activated models in inference

Original: [Figure]
Revised: [Figure]

Original: SOTA 82. As illustrated in Figure 6, the highest accuracy of 87.02% is obtained when the number of activated models reaches the highest value we set.
Revised: As illustrated in Figure 6, the highest accuracy of 87.02% is obtained when the number of activated models reaches the highest value we set.

Original: This indicates that the generalization performance can benefit from using more models to compose the ensemble for the final prediction.
Revised: This indicates that the generalization performance can benefit from using more models to compose the ensemble for the final prediction.

Original: However, another important finding of this result is that the marginal effect of increasing the number of models decreases as the number of activated models increases.
Revised: However, another important finding of this result is that the marginal effect of increasing the number of models decreases as the number of activated models increases.

Original: For example, when ˜ k is increased from 1 to 2, SIMPLE + gains 1.22% performance gain.
Revised: For example, when ˜ k is increased from 1 to 2, SIMPLE + gains 1.22% performance gain.

Original: In contrast, when ˜ k shifts from 8 to 10, the performance gain is only 0.11%.
Revised: In contrast, when ˜ k shifts from 8 to 10, the performance gain is only 0.11%.

Original: Therefore, it is predictable that since matching networks can provide a decent ranking of pretrained models, we can obtain promising generalization performance with limited activation models to save computational cost.
Revised: Therefore, it is predictable that since matching networks can provide a decent ranking of pretrained models, we can obtain promising generalization performance with limited activation models to save computational cost.

Original: Note that when ˜ k
Revised: Note that when ˜ k

Original: = 2 , our method has obtained a performance that exceeds the existing SOTA (MIRO + SWAD).
Revised: = 2 , our method has obtained a performance that exceeds the existing SOTA (MIRO + SWAD).

Original: A.8. A NALYSIS OF FEATURE EXTRACTOR SELECTION
Revised: A.9. A NALYSIS OF FEATURE EXTRACTOR SELECTION

Original: We perform an ablation study to verify the robustness of our matching network to the selection of feature extractors.
Revised: We perform an ablation study to verify the robustness of our matching network to the selection of feature extractors.

Original: Here we provide details for that claim in Section 4.5.
Revised: Here we provide details for that claim in Section 4.5.

Original: Settings.
Revised: Settings.

Original: In general, ResNet-based models are used as feature extractors.
Revised: In general, ResNet-based models are used as feature extractors.

Original: We compare the performance of our method here with a ResNet-based feature extractor (ResNet-34) and a more advanced network structure (EfficientNet-B7 with Noisy Student) to see if these changes result in significant performance differences.
Revised: We compare the performance of our method here with a ResNet-based feature extractor (ResNet-34) and a more advanced network structure (EfficientNet-B7 with Noisy Student) to see if these changes result in significant performance differences.

Original: Specifically, ResNet-34 and EfficientNet-B7-NS obtain ImageNet topclassification accuracy of 75.0% and 86.9%, respectively.
Revised: Specifically, ResNet-34 and EfficientNet-B7-NS obtain ImageNet topclassification accuracy of 75.0% and 86.9%, respectively.

Original: This ablation study is performed on the
Revised: This ablation study is performed on the

Original: OfficeHome dataset, with results shown in Table 5.
Revised: OfficeHome dataset, with results shown in Table 5.

Original: [Table]
Revised: [Table]

Original: As can be seen, using ResNet-34 or EfficientNet-B7-NS results in similar performance, without one dominating in all domains.
Revised: As can be seen, using ResNet-34 or EfficientNet-B7-NS results in similar performance, without one dominating in all domains.

Original: Therefore, feature extractor selection has little impact on the generalization performance of SIMPLE, showing its robustness.
Revised: Therefore, feature extractor selection has little impact on the generalization performance of SIMPLE, showing its robustness.

Original: Meanwhile, it is noted that the number of parameters of EfficientNet-B7-NS is three times higher than that of ResNet-34.
Revised: Meanwhile, it is noted that the number of parameters of EfficientNet-B7-NS is three times higher than that of ResNet-34.

Original: Therefore, we choose ResNet-34 because it can provide good performance with less computation cost.
Revised: Therefore, we choose ResNet-34 because it can provide good performance with less computation cost.

Original: A. T RAINING AND INFERENCE COST COMPARISON
Revised: A.9. A NALYSIS OF LABEL SPACE ADAPTER TRAINING

Revised: As detailed in Section 3.3, the label space adapter of SIMPLE is trained by two losses, L ens ( ψ, ρ ) and L adapter , and the training process will be influenced by the ensemble weights w i .

Revised: In this section, we analyze and verify whether such an influence affects the training of adapters.

Revised: Note that, we train the same adapter for all the pretrained models from the same pretraining domain, which is more efficient (than training different adapters for different pretrained models) and also avoids unexpected training instability.

Revised: When considering the large model pool with various models from different pretraining domains (e.g., Model Pool-B), we believe the influence is acceptable or even beneficial since: (1) adapters will observe all the training samples, so their training is sufficient; (2) since the number of parameters of adapters is relatively small, the training of adapters will not have a significant impact on the final performance; (3) although the training of adapters is influenced by the ensemble weights, it is unbiased because the ensemble weights are optimized according to the final objective.

Revised: Furthermore, we conduct experiments to verify whether training label space adapters under the influence of ensemble weights will affect the training of adapters or degrade the performance.

Revised: Settings.

Revised: We compare single model performance with and without the influence of ensemble weights on the corresponding label space adapter.

Revised: For no influence on adapters , we separately train individual adapters for each model.

Revised: We select several models which are assigned with small ensemble weights in SIMPLE.

Revised: The evaluation is on ‘ art ’ domain of the OfficeHome dataset.

Revised: Figure 7 shows the performance difference between (1) a model with the adapter trained with and without ensemble weights (blue bars); and (2) our method SIMPLE and the SOTA baseline for reference (orange bar).

Revised: As can be seen, (1) single model performance does not significantly degrade under the influence of ensemble weights (even less than the difference between our method SIMPLE and the existing SOTA); (2) moreover, for some models in Figure 7, training a shared adapter (i.e., influenced by the ensemble weights) can make a single model perform even better.

Revised: The possible reason is that training a shared adapter for multiple models avoids overfitting, leading to better generalization.

Revised: A. T RAINING AND INFERENCE COST COMPARISON

Original: We compare the training and inference costs of SIMPLE with general DG methods, including SOTA and methods that use ensemble learning.
Revised: We compare the training and inference costs of SIMPLE with general DG methods, including SOTA and methods that use ensemble learning.

Original: Specifically, for training costs, we evaluate the number of learnable parameters and the overall training time.
Revised: Specifically, for training costs, we evaluate the number of learnable parameters and the overall training time.

Original: And for inference, we compute GFLOPs.
Revised: And for inference, we compute GFLOPs.

Original: To fairly compare training costs, we run experiments of ERM, SWAD, and SIMPLE on a single Nvidia
Revised: To fairly compare training costs, we run experiments of ERM, SWAD, and SIMPLE on a single Nvidia

Original: Tesla V100 and compare their overall back-propagation time from the start of training to the end (or early-stop).
Revised: Tesla V100 and compare their overall back-propagation time from the start of training to the end (or early-stop).

Original: And based on the statistics of ERM and SWAD, we estimate the training time for EoA and MIRO, respectively.
Revised: And based on the statistics of ERM and SWAD, we estimate the training time for EoA and MIRO, respectively.

Original: The results are shown in Table 6.
Revised: The results are shown in Table 6.

Original: Training cost comparison.
Revised: Training cost comparison.

Original: As shown in Table 2, training SIMPLE paradigm uses noticeably less time.
Revised: As shown in Table 2, training SIMPLE paradigm uses noticeably less time.

Original: SIMPLE + takes only 0.1% of the time of ERM on PACS.
Revised: SIMPLE + takes only 0.1% of the time of ERM on PACS.

Original: The significant training time advantage of the method and its surpassing performance suggest that SIMPLE is an effective and efficient paradigm for domain generalization.
Revised: The significant training time advantage of the method and its surpassing performance suggest that SIMPLE is an effective and efficient paradigm for domain generalization.

Original: Inference cost comparison.
Revised: Inference cost comparison.

Original: Although ensemble methods like EoA and SEDGE achieve better generalization performance at the cost of higher inference FLOPs, SEDGE still manages to save a large amount of inference cost compared to the previous best ensemble model (half of the inference
Revised: Although ensemble methods like EoA and SEDGE achieve better generalization performance at the cost of higher inference FLOPs, SEDGE still manages to save a large amount of inference cost compared to the previous best ensemble model (half of the inference

Original: FLOPs compared to EoA).
Revised: FLOPs compared to EoA).

Original: This is because SIMPLE only selects models with the highest  k ( < K ) ensemble weights.
Revised: This is because SIMPLE only selects models with the highest ˜ k ( < K ) ensemble weights.

Original: Therefore, only  k of K models are activated for inference per sample, which reduces the inference cost to a large extent.
Revised: Therefore, only ˜ k of K models are activated for inference per sample, which reduces the inference cost to a large extent.

Original: In addition, compared to SOTA (MIRO + SWAD), which uses a larger network architecture to achieve better results than using ResNet-50, SIMPLE +  keeps the inference cost using significantly lower GFLOPs than SOTA uses to obtain the new SOTA results.
Revised: In addition, compared to SOTA (MIRO + SWAD), which uses a larger network architecture to achieve better results than using ResNet-50, SIMPLE + [Figure] keeps the inference cost using significantly lower GFLOPs than SOTA uses to obtain the new SOTA results.

Original: [Table]
Revised: [Table]

Original: A. M ATCHING PREFERENCE ANALYSIS
Revised: A. M ATCHING PREFERENCE ANALYSIS

Original: SIMPLE shows excellent generalization performance in the case of dispatching only fixed pretrained models to predict each OOD sample.
Revised: SIMPLE shows excellent generalization performance in the case of dispatching only fixed pretrained models to predict each OOD sample.

Original: Thus, we are curious about its matching preference, that is, which models are typically dispatched in specific domains.
Revised: Thus, we are curious about its matching preference, that is, which models are typically dispatched in specific domains.

Original: This could also provide insights into which types of pretrained models might be more suitable for certain domains.
Revised: This could also provide insights into which types of pretrained models might be more suitable for certain domains.

Original: Settings.
Revised: Settings.

Original: We attempt to analyze the preferences of our approach for network architectures and pretraining datasets.
Revised: We attempt to analyze the preferences of our approach for network architectures and pretraining datasets.

Original: To do so, we first measure the importance of the pretrained models and then perform a refined analysis of their architectures and pretraining datasets.
Revised: To do so, we first measure the importance of the pretrained models and then perform a refined analysis of their architectures and pretraining datasets.

Original: Specifically, by taking the model dispatcher trained by SIMPLE, we compute the sum of the ensemble weights assigned to each pretrained model as its importance measure for ranking.
Revised: Specifically, by taking the model dispatcher trained by SIMPLE, we compute the sum of the ensemble weights assigned to each pretrained model as its importance measure for ranking.

Original: Then, we classify these pretrained models according to the basic architecture and the pretraining dataset.
Revised: Then, we classify these pretrained models according to the basic architecture and the pretraining dataset.

Original: Treating the ranking of the pretrained models as the ranking of the associated types, we can measure the importance of each type by calculating the mean reciprocal rank (MRR).
Revised: Treating the ranking of the pretrained models as the ranking of the associated types, we can measure the importance of each type by calculating the mean reciprocal rank (MRR).

Original: Analysis of network architecture.
Revised: Analysis of network architecture.

Original: Figure10, 11, 12, 13,  and 14 show the raw rankings of the pretrained models on different datasets.
Revised: Figure 13, 14, 15, 16, and 17 show the raw rankings of the pretrained models on different datasets.

Original: From the ranking information, we can see that SIMPLE assigns markedly different pretrained models to samples from different domains.
Revised: From the ranking information, we can see that SIMPLE assigns markedly different pretrained models to samples from different domains.

Original: This may imply that these domains, as shown in Figure 14, may differ widely from each other and thus need to be handled by varying combinations of pretrained models.
Revised: This may imply that these domains, as shown in Figure 17, may differ widely from each other and thus need to be handled by varying combinations of pretrained models.

Original: We then classify these pretrained models based on their basic architecture, i.e., CNN-based, ViT-based, and MLP-based, with their MRR values over different datasets and domains shown in Figure 8.
Revised: We then classify these pretrained models based on their basic architecture, i.e., CNN-based, ViT-based, and MLP-based, with their MRR values over different datasets and domains shown in Figure 9.

Original: It can be seen that ViT-based models are dispatched more frequently than CNN-based models in most domains, with the exception of 6 domains.
Revised: It can be seen that ViT-based models are dispatched more frequently than CNN-based models in most domains, with the exception of 6 domains.

Original: Observations over these domains suggests that CNN-based models tend to be used more for real images (e.g., photo), while ViT-based models are chosen to handle stylistic or textural variations (e.g., sketch).[Figure]
Revised: Observations over these domains suggests that CNN-based models tend to be used more for real images (e.g., photo), while ViT-based models are chosen to handle stylistic or textural variations (e.g., sketch).

Original: Analysis of pretraining datasets.
Revised: Analysis of pretraining datasets.

Original: We then analyze on the pretraining datasets, including ImageNet1k, ImageNet-21k 6 , CLIP (Radford et al., 2021), and SWAG (Singh et al., 2022), used for pretrained models.
Revised: We then analyze on the pretraining datasets, including ImageNet1k, ImageNet-21k 8 , CLIP (Radford et al., 2021), and SWAG (Singh et al., 2022), used for pretrained models.

Original: The MRR results of these different pretraining datasets on five datasets are shown in Figure 9, with scores grouped by domain.
Revised: The MRR results of these different pretraining datasets on five datasets are shown in Figure 10, with scores grouped by domain.

Original: On one hand, it can be seen that our method shows a preference for SWAG, CLIP, and ImageNet-21k over ImageNet-1k dataset .
Revised: On one hand, it can be seen that our method shows a preference for SWAG, CLIP, and ImageNet-21k over ImageNet-1k dataset .

Original: This preference can be supported by recent studies finding that pretraining on ImageNet-1k is insufficient for generalization (Kumar et al., 2022) and that changing the dataset may improve generalization performance (Kim et al., 2022).
Revised: This preference can be supported by recent studies finding that pretraining on ImageNet-1k is insufficient for generalization (Kumar et al., 2022) and that changing the dataset may improve generalization performance (Kim et al., 2022).

Original: On the other hand, despite this preference, SIMPLE dispatches different models for each domain (e.g. on TerraIncognita-L100 neither SWAG nor CLIP is chosen much, but rather ImageNet-pretrained models are used), suggesting that there is still no free lunch for DG in the selection of pretraining datasets .
Revised: On the other hand, despite this preference, SIMPLE dispatches different models for each domain (e.g. on TerraIncognita-L100 neither SWAG nor CLIP is chosen much, but rather ImageNet-pretrained models are used), suggesting that there is still no free lunch for DG in the selection of pretraining datasets .

Original: A r t
Revised: A r t

Original: C a r t oo n P
Revised: C a r t oo n P

Original: h
Revised: h

Original: o t
Revised: o t

Original: o S k e t
Revised: o S k e t

Original: c h
Revised: c h

Original: C a l t e c h
Revised: C a l t e c h

Original: L a b e l M e S U N 09 V O C
Revised: L a b e l M e S U N 09 V O C

Revised: [Figure]

Original: A r t C li p a r t P r o d u c t P h
Revised: A r t C li p a r t P r o d u c t P h

Original: o t o L 100 L 38 L 43 L 46 C li p a r t
Revised: o t o L 100 L 38 L 43 L 46 C li p a r t

Original: I n f o g r a p
Revised: I n f o g r a p

Original: h
Revised: h

Original: i c
Revised: i c

Original: P a i n t i n g
Revised: P a i n t i n g

Original: Q u i c k D r a w P h
Revised: Q u i c k D r a w P h

Original: o t o S k e t
Revised: o t o S k e t

Original: c h 0.
Revised: c h 0.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i
Revised: i

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: A. L EVERAGING MATCHING PREFERENCE AS TRANSFERABILITY MEASUREMENT

Revised: In this section, we leverage the matching preferences of the learned model dispatcher to optimize the model pool size and study the performance of SIMPLE for different model pool sizes.

Revised: Specifically, since the model dispatcher of SIMPLE learns to match the best suitable models for unseen samples at a meta-level, its matching preference over unseen domains (without accessing ground truth labels) can be regarded as a measurement of model transferability on these unseen samples.

Revised: Therefore, we can further optimize the model pool size used for each domain in a two-stage manner: (1) first learn with a large model pool on source domains to generate matching preferences; and (2) then reconstruct a smaller model pool to include models preferred by our dispatcher.

Revised: Settings.

Revised: We experiment this two-manner training on PACS and OfficeHome datasets, using different reconstructed model pool size in the second stage.

Revised: Note that we construct a model pool specific to each domain in a dataset, based on aggregated matching preferences for all samples of the unseen domain.

Revised: The results of PACS and OfficeHome datasets are shown in Figure 11 and 12, respectively.

Revised: As can be seen, given that SIMPLE can automatically match models more suitable for transferring to unseen domains, we can actually go beyond the existing SOTA approach even using a model pool with only two models (while the best single model fails to outperform since there is No Free Lunch).

Revised: And when using a smaller model pool, SIMPLE can even further improve the performance of the

Revised: OfficeHome dataset.

Revised: [Figure]

Revised: [Figure]

Revised: a g e a cc u r a c y

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: timm ssl resnext50 32x4d 4062.
Revised: timm ssl resnext50 32x4d 4062.

Original: timm swin base patch4 window7 102.
Revised: timm swin base patch4 window7 102.

Original: timm swin large patch4 window7 215.
Revised: timm swin large patch4 window7 215.

Original: timm swin small patch4 window7 61.
Revised: timm swin small patch4 window7 61.

Original: timm swin tiny patch4 window7 40.
Revised: timm swin tiny patch4 window7 40.

Original: timm swsl resnet  1734.
Revised: timm swsl resnet  1734.

Original: timm swsl resnet  3921.
Revised: timm swsl resnet  3921.

Original: timm swsl resnext50 32x4d 4062.
Revised: timm swsl resnext50 32x4d 4062.

Original: timm tf efficientnet b  351.
Revised: timm tf efficientnet b  351.

Original: timm tf efficientnet b0 ap 351.
Revised: timm tf efficientnet b0 ap 351.

Original: timm tf efficientnet b0 ns 351.
Revised: timm tf efficientnet b0 ns 351.

Original: timm tf efficientnet cc b0 4e 154. timm tf efficientnet cc b0 8e 154.
Revised: timm tf efficientnet cc b0 4e 154. timm tf efficientnet cc b0 8e 154.

Original: timm tf efficientnet es 1513.
Revised: timm tf efficientnet es 1513.

Original: timm tf efficientnet lite  350.
Revised: timm tf efficientnet lite  350.

Original: timm tf mixnet l 492.
Revised: timm tf mixnet l 492.

Original: timm tf mixnet m 295.
Revised: timm tf mixnet m 295.

Original: timm tf mixnet s 214.
Revised: timm tf mixnet s 214.

Original: timm tf mobilenetv3 large 137.
Revised: timm tf mobilenetv3 large 137.

Original: timm tf mobilenetv3 large 195.
Revised: timm tf mobilenetv3 large 195.

Original: timm tf mobilenetv3 large minimal 190.
Revised: timm tf mobilenetv3 large minimal 190.

Original: timm tf mobilenetv3 small 33.
Revised: timm tf mobilenetv3 small 33.

Original: timm tf mobilenetv3 small 46.
Revised: timm tf mobilenetv3 small 46.

Original: timm tf mobilenetv3 small minimal 42.
Revised: timm tf mobilenetv3 small minimal 42.

Original: timm tnt s patch16 33.
Revised: timm tnt s patch16 33.

Original: timm tv densenet
Revised: timm tv densenet

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: timm twins pcpvt small 317.
Revised: timm twins pcpvt small 317.

Original: timm twins svt base 534.
Revised: timm twins svt base 534.

Original: timm twins svt large 940.
Revised: timm twins svt large 940.

Original: timm twins svt small 192.
Revised: timm twins svt small 192.

Original: timm visformer small 4532.
Revised: timm visformer small 4532.

Original: timm vit base patch16 192.
Revised: timm vit base patch16 192.

Original: timm vit base patch16 224 miil 192.
Revised: timm vit base patch16 224 miil 192.

Original: timm vit base patch32 192.
Revised: timm vit base patch32 192.

Original: timm cait s24 99.
Revised: timm cait s24 99.

Original: timm cait xxs24 38.
Revised: timm cait xxs24 38.

Original: timm cait xxs36 43.
Revised: timm cait xxs36 43.

Original: timm coat lite mini 135.
Revised: timm coat lite mini 135.

Original: timm coat lite small 170.
Revised: timm coat lite small 170.

Original: timm coat lite tiny 107.
Revised: timm coat lite tiny 107.

Original: timm coat mini 284.
Revised: timm coat mini 284.

Original: timm coat tiny 206.
Revised: timm coat tiny 206.

Original: timm convit base 192.
Revised: timm convit base 192.

Original: timm convit small 88.
Revised: timm convit small 88.

Original: timm convit tiny 32.
Revised: timm convit tiny 32.

Original: timm cspresnext  2942.
Revised: timm cspresnext  2942.

Original: timm deit base distilled patch16 192.
Revised: timm deit base distilled patch16 192.

Original: timm deit base patch16 192.
Revised: timm deit base patch16 192.

Original: timm deit small distilled patch16 76.
Revised: timm deit small distilled patch16 76.

Original: timm deit small patch16 75.
Revised: timm deit small patch16 75.

Original: timm deit tiny distilled patch16 33.
Revised: timm deit tiny distilled patch16 33.

Original: timm deit tiny patch16 32.
Revised: timm deit tiny patch16 32.

Original: timm densenet  2703.
Revised: timm densenet  2703.

Original: timm densenet  3204.
Revised: timm densenet  3204.

Original: timm densenet  4092.
Revised: timm densenet  4092.

Original: timm densenetblur121d 2931.
Revised: timm densenetblur121d 2931.

Original: timm dla  2927.
Revised: timm dla  2927.

Original: timm dla46 c 556.
Revised: timm dla46 c 556.

Original: timm dla46x c 518.
Revised: timm dla46x c 518.

Original: timm dla  4059.
Revised: timm dla  4059.

Original: timm dla60 res2net 3955.
Revised: timm dla60 res2net 3955.

Original: timm dla60 res2next 3323.
Revised: timm dla60 res2next 3323.

Original: timm dla60x 3380.
Revised: timm dla60x 3380.

Original: timm dla60x c 565.
Revised: timm dla60x c 565.

Original: timm dm nfnet f  45.
Revised: timm dm nfnet f  45.

Original: timm dpn  2218.
Revised: timm dpn  2218.

Original: timm dpn68b 2218.
Revised: timm dpn68b 2218.

Original: timm eca nfnet l  2.
Revised: timm eca nfnet l  2.

Original: timm ecaresnet50d 4151.
Revised: timm ecaresnet50d 4151.

Original: timm ecaresnet50d pruned 2413.
Revised: timm ecaresnet50d pruned 2413.

Original: timm ecaresnet101d pruned 3313.
Revised: timm ecaresnet101d pruned 3313.

Original: timm ecaresnetlight 3916.
Revised: timm ecaresnetlight 3916.

Original: timm efficientnet b  368.
Revised: timm efficientnet b  368.

Original: timm efficientnet b  543.
Revised: timm efficientnet b  543.

Original: timm efficientnet es 1705. timm efficientnet es pruned 1705.
Revised: timm efficientnet es 1705. timm efficientnet es pruned 1705.

Original: timm efficientnet lite  367.
Revised: timm efficientnet lite  367.

Original: timm ese vovnet19b dw 1271.
Revised: timm ese vovnet19b dw 1271.

Original: timm fbnetc 367.
Revised: timm fbnetc 367.

Original: timm gernet m 2865.
Revised: timm gernet m 2865.

Original: timm gernet s 709.
Revised: timm gernet s 709.

Original: timm ghostnet 140.
Revised: timm ghostnet 140.

Original: timm gluon resnet18 v1b 1734.
Revised: timm gluon resnet18 v1b 1734.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: timm gluon resnet50 v1c 4151. timm gluon resnet50 v1d 4151.
Revised: timm gluon resnet50 v1c 4151. timm gluon resnet50 v1d 4151.

Original: timm gluon resnext50 32x4d 4062.
Revised: timm gluon resnext50 32x4d 4062.

Original: timm gluon seresnext50 32x4d 4064.
Revised: timm gluon seresnext50 32x4d 4064.

Original: timm gmixer 24 78.
Revised: timm gmixer 24 78.

Original: timm gmlp s16 55.
Revised: timm gmlp s16 55.

Original: timm hardcorenas a 212.
Revised: timm hardcorenas a 212.

Original: timm hardcorenas b 238.
Revised: timm hardcorenas b 238.

Original: timm hardcorenas c 256.
Revised: timm hardcorenas c 256.

Original: timm hardcorenas d 273.
Revised: timm hardcorenas d 273.

Original: timm hardcorenas e 321.
Revised: timm hardcorenas e 321.

Original: timm hardcorenas f 324.
Revised: timm hardcorenas f 324.

Original: timm hrnet w  4116.
Revised: timm hrnet w  4116.

Original: timm hrnet w18 small 1538.
Revised: timm hrnet w18 small 1538.

Original: timm hrnet w18 small v  2494.
Revised: timm hrnet w18 small v  2494.

Original: timm legacy seresnet  1734.
Revised: timm legacy seresnet  1734.

Original: timm legacy seresnet  3501.
Revised: timm legacy seresnet  3501.

Original: timm legacy seresnet  3701.
Revised: timm legacy seresnet  3701.

Original: timm legacy seresnext26 32x4d 2375.
Revised: timm legacy seresnext26 32x4d 2375.

Original: timm legacy seresnext50 32x4d 4064.
Revised: timm legacy seresnext50 32x4d 4064.

Original: timm levit 55.
Revised: timm levit 55.

Original: timm levit 128s 54.
Revised: timm levit 128s 54.

Original: timm levit 112.
Revised: timm levit 112.

Original: timm levit 194.
Revised: timm levit 194.

Original: timm levit 426.
Revised: timm levit 426.

Original: timm mixer b16 166.
Revised: timm mixer b16 166.

Original: timm mixer b16 224 miil 166.
Revised: timm mixer b16 224 miil 166.

Original: timm mixer l16 344.
Revised: timm mixer l16 344.

Original: timm mixnet l 529.
Revised: timm mixnet l 529.

Original: timm mixnet m 323.
Revised: timm mixnet m 323.

Original: timm mixnet s 228.
Revised: timm mixnet s 228.

Original: timm mixnet xl 862.
Revised: timm mixnet xl 862.

Original: timm mnasnet 299.
Revised: timm mnasnet 299.

Original: timm mobilenetv2 286.
Revised: timm mobilenetv2 286.

Original: timm mobilenetv2 110d 410.
Revised: timm mobilenetv2 110d 410.

Original: timm mobilenetv2 120d 637.
Revised: timm mobilenetv2 120d 637.

Original: timm mobilenetv2 555.
Revised: timm mobilenetv2 555.

Original: timm mobilenetv3 large 205.
Revised: timm mobilenetv3 large 205.

Original: timm mobilenetv3 large 100 miil 205.
Revised: timm mobilenetv3 large 100 miil 205.

Original: timm mobilenetv3 rw 205.
Revised: timm mobilenetv3 rw 205.

Original: timm nfnet l  13.
Revised: timm nfnet l  13.

Original: timm pit b 209.
Revised: timm pit b 209.

Original: timm pit b distilled 210.
Revised: timm pit b distilled 210.

Original: timm pit s 99.
Revised: timm pit s 99.

Original: timm pit s distilled 100.
Revised: timm pit s distilled 100.

Original: timm pit ti 39.
Revised: timm pit ti 39.

Original: timm pit ti distilled 39.
Revised: timm pit ti distilled 39.

Original: timm pit xs 61.
Revised: timm pit xs 61.

Original: timm pit xs distilled 62.
Revised: timm pit xs distilled 62.

Original: timm regnetx 189.
Revised: timm regnetx 189.

Original: timm regnetx 379.
Revised: timm regnetx 379.

Original: timm regnetx 573.
Revised: timm regnetx 573.

Original: timm regnetx 762.
Revised: timm regnetx 762.

Original: timm regnetx 1528.
Revised: timm regnetx 1528.

Original: timm regnetx 3029.
Revised: timm regnetx 3029.

Original: timm regnetx 3780.
Revised: timm regnetx 3780.

Original: timm regnety 190.
Revised: timm regnety 190.

Original: timm regnety 383.
Revised: timm regnety 383.

Original: timm regnety 573.
Revised: timm regnety 573.

Original: timm regnety 760.
Revised: timm regnety 760.

Original: timm regnety 1537.
Revised: timm regnety 1537.

Original: timm regnety 3029.
Revised: timm regnety 3029.

Original: timm regnety 3789.
Revised: timm regnety 3789.

Original: timm repvgg b  3239.
Revised: timm repvgg b  3239.

Original: timm res2net50 14w 8s 4010.
Revised: timm res2net50 14w 8s 4010.

Original: timm res2net50 26w 4s 4082.
Revised: timm res2net50 26w 4s 4082.

Original: timm res2net50 48w 2s 3989.
Revised: timm res2net50 48w 2s 3989.

Original: timm res2next  4005.
Revised: timm res2next  4005.

Original: timm resmlp 12 69.
Revised: timm resmlp 12 69.

Original: timm resmlp 12 distilled 69.
Revised: timm resmlp 12 distilled 69.

Original: timm resmlp 24 83.
Revised: timm resmlp 24 83.

Original: timm resmlp 24 distilled 83.
Revised: timm resmlp 24 distilled 83.

Original: timm resmlp 36 97.
Revised: timm resmlp 36 97.

Original: timm resmlp 36 distilled 97.
Revised: timm resmlp 36 distilled 97.

Original: timm resmlp big 24 233.
Revised: timm resmlp big 24 233.

Original: timm resmlp big 24 224 in22ft1k 233.
Revised: timm resmlp big 24 224 in22ft1k 233.

Original: timm resmlp big 24 distilled 233.
Revised: timm resmlp big 24 distilled 233.

Original: timm resnest14d 2636.
Revised: timm resnest14d 2636.

Original: timm resnest26d 3475.
Revised: timm resnest26d 3475.

Original: timm resnest50d 1s4x24d 4222.
Revised: timm resnest50d 1s4x24d 4222.

Original: timm resnest50d 4s2x40d 4201.
Revised: timm resnest50d 4s2x40d 4201.

Original: timm resnet  1734.
Revised: timm resnet  1734.

Original: timm resnet18d 1964.
Revised: timm resnet18d 1964.

Original: timm resnet  2247.
Revised: timm resnet  2247.

Original: timm resnet26d 2478.
Revised: timm resnet26d 2478.

Original: timm resnet  3501.
Revised: timm resnet  3501.

Original: timm resnet34d 3731.
Revised: timm resnet34d 3731.

Original: timm resnet  3921.
Revised: timm resnet  3921.

Original: timm resnet50d 4151.
Revised: timm resnet50d 4151.

Original: timm resnetblur  4914.
Revised: timm resnetblur  4914.

Original: timm resnetv2 50x1 bit distilled 2.
Revised: timm resnetv2 50x1 bit distilled 2.

Original: timm vit large patch16 436.
Revised: timm vit large patch16 436.

Original: timm vit large r50 s32 387.
Revised: timm vit large r50 s32 387.

Original: timm vit small patch16 75.
Revised: timm vit small patch16 75.

Original: timm vit small patch32 75.
Revised: timm vit small patch32 75.

Original: timm vit small r26 s32 57.
Revised: timm vit small r26 s32 57.

Original: timm vit tiny patch16 32.
Revised: timm vit tiny patch16 32.

Original: timm vit tiny r s16 p8 42.
Revised: timm vit tiny r s16 p8 42.

Original: pretrainedmodels alexnet 681.
Revised: pretrainedmodels alexnet 681.

Original: pretrainedmodels densenet  2732.
Revised: pretrainedmodels densenet  2732.

Original: pretrainedmodels densenet  3240.
Revised: pretrainedmodels densenet  3240.

Original: pretrainedmodels densenet  4139.
Revised: pretrainedmodels densenet  4139.

Original: pretrainedmodels dpn  2241.
Revised: pretrainedmodels dpn  2241.

Original: pretrainedmodels nasnetamobile 551.
Revised: pretrainedmodels nasnetamobile 551.

Original: pretrainedmodels resnet  1734.
Revised: pretrainedmodels resnet  1734.

Original: pretrainedmodels resnet  3501.
Revised: pretrainedmodels resnet  3501.

Original: pretrainedmodels resnet  3921.
Revised: pretrainedmodels resnet  3921.

Original: pretrainedmodels se resnet  3707.
Revised: pretrainedmodels se resnet  3707.

Original: swag regnety 16gf in1k 15223.
Revised: swag regnety 16gf in1k 15223.

Original: swag regnety 32gf in1k 30849.
Revised: swag regnety 32gf in1k 30849.

Original: timm beit base patch16 171.
Revised: timm beit base patch16 171.

Original: timm beit large patch

Revised: timm beit large patch

Original: [Table]
Revised: [Table]

Original: timm convnext base
Revised: timm convnext base

Original: [Table]
Revised: [Table]

Original: timm convnext
Revised: timm convnext

Original: [Table] [Table]
Revised: [Table] [Table]

Original: timm convnext 
Revised: timm convnext base

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: timm convnext large
Revised: timm convnext large

Original: [Table]
Revised: [Table]

Original: timm convnext large
Revised: timm convnext large

Original: [Table]

Original: [Table]
Revised: [Table]

Original: timm convnext large in22ft1k 
Revised: timm convnext large in22ft1k 1205.

Original: [Table]

Original: timm convnext nano 2340. timm convnext nano hnf 2340.
Revised: timm convnext nano 2340. timm convnext nano hnf 2340.

Original: timm convnext nano ols 2386.
Revised: timm convnext nano ols 2386.

Original: timm convnext small 392.
Revised: timm convnext small 392.

Original: timm convnext small 384 in22ft1k 392. timm convnext small in22ft1k 392.
Revised: timm convnext small 384 in22ft1k 392. timm convnext small in22ft1k 392.

Original: timm convnext tiny 307.
Revised: timm convnext tiny 307.

Original: timm convnext tiny 384 in22ft1k 307.
Revised: timm convnext tiny 384 in22ft1k 307.

Original: timm convnext tiny hnf 4261.
Revised: timm convnext tiny hnf 4261.

Original: timm convnext tiny in22ft1k 307.
Revised: timm convnext tiny in22ft1k 307.

Original: timm convnext xlarge 384 in22ft1k 1980.
Revised: timm convnext xlarge 384 in22ft1k 1980.

Original: timm convnext xlarge in22ft1k 1980.
Revised: timm convnext xlarge in22ft1k 1980.

Original: MAE vit base patch  192.
Revised: MAE vit base patch  192.

Original: MAE vit large patch  436.
Revised: MAE vit large patch  436.

Original: MAE vit huge patch  785.
Revised: MAE vit huge patch  785.

Original: swag vit b  164.
Revised: swag vit b  164.

Original: swag vit l  339.
Revised: swag vit l  339.

Original: swag vit h  584.
Revised: swag vit h  584.

Original: swag regnety 16gf 15220.
Revised: swag regnety 16gf 15220.

Original: swag regnety 32gf 30845.
Revised: swag regnety 32gf 30845.

Original: timm vit base patch16 224 miil in21k 199.
Revised: timm vit base patch16 224 miil in21k 199.

Original: timm mixer b16 224 miil in21k 174.
Revised: timm mixer b16 224 miil in21k 174.

Original: timm mobilenetv3 large 100 miil in21k 205.
Revised: timm mobilenetv3 large 100 miil in21k 205.

Original: timm swin base patch4 window7 224 in22k 122.
Revised: timm swin base patch4 window7 224 in22k 122.

Original: timm swin large patch4 window7 224 in22k 245.
Revised: timm swin large patch4 window7 224 in22k 245.

Original: timm beit base patch16 224 in22k 187.
Revised: timm beit base patch16 224 in22k 187.

Original: timm beit large patch16 224 in22k 384.
Revised: timm beit large patch16 224 in22k 384.

Original: timm convnext base in22k 636.
Revised: timm convnext base in22k 636.

Original: timm convnext large in22k 1235.
Revised: timm convnext large in22k 1235.

Original: timm convnext small in22k 407.
Revised: timm convnext small in22k 407.

Original: timm convnext tiny in22k 322.
Revised: timm convnext tiny in22k 322.

Original: timm convnext xlarge in22k 2021.
Revised: timm convnext xlarge in22k 2021.

Original: timm resnetv2 50x1 bitm in21k 42.
Revised: timm resnetv2 50x1 bitm in21k 42.

Original: timm resnetv2 50x3 bitm in21k 128.
Revised: timm resnetv2 50x3 bitm in21k 128.

Original: timm resnetv2 101x1 bitm in21k 42.
Revised: timm resnetv2 101x1 bitm in21k 42.

Original: timm resnetv2 152x2 bitm in21k 85.
Revised: timm resnetv2 152x2 bitm in21k 85.

Original: timm vit base patch16 224 in21k 207.
Revised: timm vit base patch16 224 in21k 207.

Original: timm vit base patch32 224 in21k 207.
Revised: timm vit base patch32 224 in21k 207.

Original: timm vit base r50 s16 224 in21k 244.
Revised: timm vit base r50 s16 224 in21k 244.

Original: timm mixer b16 224 in21k 182.
Revised: timm mixer b16 224 in21k 182.

Original: timm mixer l16 224 in21k 365.
Revised: timm mixer l16 224 in21k 365.

Original: timm vit huge patch14 224 in21k 811.
Revised: timm vit huge patch14 224 in21k 811.

Original: timm vit large patch16 224 in21k 456. timm vit large patch32 224 in21k 456.
Revised: timm vit large patch16 224 in21k 456. timm vit large patch32 224 in21k 456.

Original: timm vit large r50 s32 224 in21k 407.
Revised: timm vit large r50 s32 224 in21k 407.

Original: timm vit small patch16 224 in21k 83.
Revised: timm vit small patch16 224 in21k 83.

Original: timm vit small patch32 224 in21k 83.
Revised: timm vit small patch32 224 in21k 83.

Original: timm vit small r26 s32 224 in21k 65.
Revised: timm vit small r26 s32 224 in21k 65.

Original: timm vit tiny patch16 224 in21k 36.
Revised: timm vit tiny patch16 224 in21k 36.

Original: timm vit tiny r s16 p8 224 in21k 45.
Revised: timm vit tiny r s16 p8 224 in21k 45.

Original: clip
Revised: clip

Original: ViT-Barbitrary 188.
Revised: ViT-Barbitrary 188.

Original: clip
Revised: clip

Original: ViT-Barbitrary 188.
Revised: ViT-Barbitrary 188.