Original: Empirical Study on Optimizer Selection for Out-of-Distribution Generalization
Revised: Empirical Study on Optimizer Selection for Out-of-Distribution Generalization

Original: Abstract
Revised: Abstract

Original: Modern deep learning systems are fragile and do not generalize well under distribution shifts.
Revised: Modern deep learning systems are fragile and do not generalize well under distribution shifts.

Original: While much promising work has been accomplished to address these concerns, a systematic study of the role of optimizers and their out-of-distribution generalization performance has not been undertaken.
Revised: While much promising work has been accomplished to address these concerns, a systematic study of the role of optimizers and their out-of-distribution generalization performance has not been undertaken.

Original: In this study, we examine the performance of popular first-order optimizers for different classes of distributional shift under empirical risk minimization and invariant risk minimization.
Revised: In this study, we examine the performance of popular first-order optimizers for different classes of distributional shift under empirical risk minimization and invariant risk minimization.

Original: We address the problem settings for image and text classification using DomainBed, WILDS, and Backgrounds Challenge as out-of-distribution datasets for the exhaustive study.
Revised: We address the problem settings for image and text classification using DomainBed, WILDS, and Backgrounds Challenge as out-of-distribution datasets for the exhaustive study.

Original: We search over a wide range of hyperparameters and examine the classification accuracy (in-distribution and out-of-distribution) for over 20,000 models.
Revised: We search over a wide range of hyperparameters and examine the classification accuracy (in-distribution and out-of-distribution) for over 20,000 models.

Original: We arrive at the following findings: i) contrary to conventional wisdom, adaptive optimizers (e.g., Adam) perform worse than non-adaptive optimizers (e.g., SGD, momentumbased SGD), ii) in-distribution performance and out-of-distribution performance exhibit three types of behavior depending on the dataset – linear returns, increasing returns, and diminishing returns.
Revised: We arrive at the following findings: i) contrary to conventional wisdom, adaptive optimizers (e.g., Adam) perform worse than non-adaptive optimizers (e.g., SGD, momentumbased SGD), ii) in-distribution performance and out-of-distribution performance exhibit three types of behavior depending on the dataset – linear returns, increasing returns, and diminishing returns.

Original: We believe these findings can help practitioners choose the right optimizer and know what behavior to expect.
Revised: We believe these findings can help practitioners choose the right optimizer and know what behavior to expect.

Original: The code is available at https://github.com/Hiroki11x/Optimizer_Comparison_OOD .
Revised: The code is available at https://github.com/Hiroki11x/Optimizer_Comparison_OOD .

Original:  Introduction
Revised:  Introduction

Original: Selecting an appropriate optimizer is crucial for the successful training of deep neural networks.
Revised: Selecting an appropriate optimizer is crucial for the successful training of deep neural networks.

Original: For example, the optimizer influences training speed, stability and generalization performance.
Revised: For example, the optimizer influences training speed, stability and generalization performance.

Original: Several studies have compared a variety of optimizers and investigated their influence on trainability and generalizability [Wil+17; SBH19; Cho+19].
Revised: Several studies have compared a variety of optimizers and investigated their influence on trainability and generalizability [Wil+17; SBH19; Cho+19].

Original: Some studies concluded that non-adaptive optimizers provide better generalization
Revised: Some studies concluded that non-adaptive optimizers provide better generalization

Original: [Wil+17; BH18], while others countered that optimizer selection does not affect generalization performance [SBH19; SSH20].
Revised: [Wil+17; BH18], while others countered that optimizer selection does not affect generalization performance [SBH19; SSH20].

Original: Previous conflicting reports have relied on the lack of hyperparameter search, and exhaustive experiments have shown that adaptive optimization methods can sometimes outperform non-adaptive methods in terms of generalization performance when tuned properly [Cho+19].
Revised: Previous conflicting reports have relied on the lack of hyperparameter search, and exhaustive experiments have shown that adaptive optimization methods can sometimes outperform non-adaptive methods in terms of generalization performance when tuned properly [Cho+19].

Original: Although these studies have made substantial progress to improve our understanding of optimizer characteristics, they are based on a common assumption of supervised learning in which the training and test data are drawn from the same distribution.
Revised: Although these studies have made substantial progress to improve our understanding of optimizer characteristics, they are based on a common assumption of supervised learning in which the training and test data are drawn from the same distribution.

Original: In real-world applications, however, it is often the case that the test data obey a distribution different from the training data, and this distributional shift violates the typical independent and identically distributed (i.i.d.) assumption for learning [NAN21].
Revised: In real-world applications, however, it is often the case that the test data obey a distribution different from the training data, and this distributional shift violates the typical independent and identically distributed (i.i.d.) assumption for learning [NAN21].

Original: Comparing the generalization performance under this distributional shift, known as out-ofdistribution (OOD) generalization [She+21], among different optimizers is of great interest in theory and in practice.
Revised: Comparing the generalization performance under this distributional shift, known as out-ofdistribution (OOD) generalization [She+21], among different optimizers is of great interest in theory and in practice.

Original: Because our objective is to investigate the impact of commonly used optimizers in the OOD problem setting, we target five of the most popular and standard optimizers that have been used and studied in recent years [SSH21].
Revised: Because our objective is to investigate the impact of commonly used optimizers in the OOD problem setting, we target five of the most popular and standard optimizers that have been used and studied in recent years [SSH21].

Original: In this work, we evaluate the OOD generalization performance ontypical OOD benchmarks using different optimizers, including stochastic gradient descent (SGD),
Revised: In this work, we evaluate the OOD generalization performance ontypical OOD benchmarks using different optimizers, including stochastic gradient descent (SGD),

Original: Momentum SGD [Pol64], and Nesterov accelerated gradient (NAG, also known as Nesterov’s momentum)
Revised: Momentum SGD [Pol64], and Nesterov accelerated gradient (NAG, also known as Nesterov’s momentum)

Original: [Nes03] in the family of non-adaptive methods as well as RMSProp [TH12] and Adam [KB15] in the family of adaptive optimizers.
Revised: [Nes03] in the family of non-adaptive methods as well as RMSProp [TH12] and Adam [KB15] in the family of adaptive optimizers.

Original: We use DomainBed (which includes seven image datasets)
Revised: We use DomainBed (which includes seven image datasets)

Original: [GL21], Backgrounds Challenge dataset [Xia+21], Amazon-WILDS, and CivilCommentsWILDS [Koh+21] for the benchmarks.
Revised: [GL21], Backgrounds Challenge dataset [Xia+21], Amazon-WILDS, and CivilCommentsWILDS [Koh+21] for the benchmarks.

Original: We train deep neural networks using either Empirical Risk
Revised: We train deep neural networks using either Empirical Risk

Original: Minimization (ERM) or Invariant Risk Minimization (IRM)
Revised: Minimization (ERM) or Invariant Risk Minimization (IRM)

Original: [Arj+19] with various optimizers.
Revised: [Arj+19] with various optimizers.

Original: We conduct an exhaustive hyperparameter search to obtain hyperparameters that give good in-domain validation accuracy for each optimizer.
Revised: We conduct an exhaustive hyperparameter search to obtain hyperparameters that give good in-domain validation accuracy for each optimizer.

Original: We then test the models on the above OOD test sets.
Revised: We then test the models on the above OOD test sets.

Original: An advantage of our approach is that we consider a wide range of hyperparameter configurations and datasets, including image classification and NLP problem settings.
Revised: An advantage of our approach is that we consider a wide range of hyperparameter configurations and datasets, including image classification and NLP problem settings.

Original: Given that the parameter search space was different from that of previous studies [Wil+17; SBH19; Cho+19], exploring hyperparameters as comprehensively as possible is crucial for a reliable investigation.
Revised: Given that the parameter search space was different from that of previous studies [Wil+17; SBH19; Cho+19], exploring hyperparameters as comprehensively as possible is crucial for a reliable investigation.

Original: Therefore, we follow Choi et al.
Revised: Therefore, we follow Choi et al.

Original: [Cho+19], who most exhaustively explored hyperparameters for parameter searches.
Revised: [Cho+19], who most exhaustively explored hyperparameters for parameter searches.

Original: We further checked the soundness of our experiments.
Revised: We further checked the soundness of our experiments.

Original: Using the same optimizer as the existing benchmark, we explored more hyperparameter configurations and recorded results that exceeded the best-reported performance.
Revised: Using the same optimizer as the existing benchmark, we explored more hyperparameter configurations and recorded results that exceeded the best-reported performance.

Original: In summary, our contributions are as follows:
Revised: In summary, our contributions are as follows:

Original: • We design and perform a comparison of the effect of optimizers on OOD generalization on a number of OOD benchmarks.
Revised: • We design and perform a comparison of the effect of optimizers on OOD generalization on a number of OOD benchmarks.

Original: To the best of our knowledge, we are the first to consider such a wide variety and scale of datasets.
Revised: To the best of our knowledge, we are the first to consider such a wide variety and scale of datasets.

Original: Also, We conduct an empirical study using a wide range of hyperparameter configurations, examining over 20,000 models, evaluating their performance, and measuring the performance gap when moving from the in-distribution test set to the OOD test set.
Revised: Also, We conduct an empirical study using a wide range of hyperparameter configurations, examining over 20,000 models, evaluating their performance, and measuring the performance gap when moving from the in-distribution test set to the OOD test set.

Original: • We demonstrate that, for the exhaustive image classification and NLP tasks, differences in the optimizer lead to differences in the OOD generalization performance.
Revised: • We demonstrate that, for the exhaustive image classification and NLP tasks, differences in the optimizer lead to differences in the OOD generalization performance.

Original: The adaptive optimizers provide more in-distribution overfitting and degrade OOD performance more than the non-adaptive optimizers within our exhausive experiments.
Revised: The adaptive optimizers provide more in-distribution overfitting and degrade OOD performance more than the non-adaptive optimizers within our exhausive experiments.

Original: • We show that the observed correlation behaviors between in-distribution performance and OOD performance can be categorized into typical patterns: linear return, diminishing return, and increasing return 3 .
Revised: • We show that the observed correlation behaviors between in-distribution performance and OOD performance can be categorized into typical patterns: linear return, diminishing return, and increasing return 3 .

Original: This result will help practitioners better understand and select optimizers.
Revised: This result will help practitioners better understand and select optimizers.

Original: Our observation is that the adaptive and non-adaptive optimizers have no significant performance difference in terms of in-distribution generalization performance.
Revised: Our observation is that the adaptive and non-adaptive optimizers have no significant performance difference in terms of in-distribution generalization performance.

Original: In contrast, the adaptive optimizers are inferior in OOD generalization performance on 8 out of 10 datasets.
Revised: In contrast, the adaptive optimizers are inferior in OOD generalization performance on 8 out of 10 datasets.

Original: In other words, in all appropriate problem settings of image and text classification, we observed that non-adaptive optimizers show superior OOD performance.
Revised: In other words, in all appropriate problem settings of image and text classification, we observed that non-adaptive optimizers show superior OOD performance.

Original: We also observe a similar trend not only for training with ERM but also with IRM.
Revised: We also observe a similar trend not only for training with ERM but also with IRM.

Original: This result is consistent with theoretical results [Zou+22] showing the drawback of adaptive optimizers such as
Revised: This result is consistent with theoretical results [Zou+22] showing the drawback of adaptive optimizers such as

Original: Adam under the i.i.d.
Revised: Adam under the i.i.d.

Original: assumption, where instead of fast convergence, they tend to fit the noise in the data. 
Revised: assumption, where instead of fast convergence, they tend to fit the noise in the data. [Table] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Original:  Preliminaries 2.1 Out-of-Distribution Generalization Datasets
Revised:  Preliminaries 2.1 Out-of-Distribution Generalization Datasets

Original: We use the following datasets to evaluate the optimizer influence on OOD generalization: DomainBed [GL21], Backgrounds Challenge [Xia+21], Amazon-WILDS [Koh+21], and CivilComments-WILDS
Revised: We use the following datasets to evaluate the optimizer influence on OOD generalization: DomainBed [GL21], Backgrounds Challenge [Xia+21], Amazon-WILDS [Koh+21] and CivilComments-WILDS

Original: [Koh+21].
Revised: [Koh+21].

Original: These datasets include both artificially created and real-world data, and the applications [Table] [Figure] d i
Revised: These datasets include both artificially created and real-world data and the applications include image classification and text classification.

Original: [Figure]

Original: t r i

Original: [Figure]

Original: u t i o n a

Original: [Figure]

Original: c

Original: [Figure]

Original: t -

Original: [Figure] f - d i s t r i [Figure] t i o n
Revised: We describe the details of these datasets in

Original: [Figure] c [Figure]
Revised: Appendix C.2.

Original: include image classification and text classification.

Original: 2.2 Model Selection Method and Evaluation Metrics
Revised: 2.2 Model Selection Method and Evaluation Metrics

Original: We follow the method of the benchmark paper for each dataset in the model selection method [GL21;
Revised: We follow the method of the benchmark paper for each dataset in the model selection method [GL21;

Original: Xia+21; Koh+21].
Revised: Xia+21; Koh+21].

Original: The OOD accuracy also follows existing studies for each dataset.
Revised: The OOD accuracy also follows existing studies for each dataset.

Original: A detailed description is provided in Appendix E.
Revised: A detailed description is provided in Appendix E.

Original:  Experiments 3.1 Experimental Overview
Revised:  Experiments 3.1 Experimental Overview

Original: Our study aims to elucidate the influence of optimizer selection under distributional shifts.
Revised: Our study aims to elucidate the influence of optimizer selection under distributional shifts.

Original: To that end, we perform image and text classification and evaluate the trained model accuracy on the benchmark datasets introduced in Section 2.1.
Revised: To that end, we perform image and text classification and evaluate the trained model accuracy on the benchmark datasets introduced in Section 2.1.

Revised: By comparing the test accuracy for in-distribution samples with

Original: By comparing the test accuracy for in-distribution samples with that for OOD samples, we can observe how the solution found by each optimizer is robust to the distributional shift.
Revised: [Figure] d i s t r i b u t i o n a [Figure] that for OOD samples, we can observe how the solution found by each optimizer is robust to the distributional shift.

Original: We investigate both ERM and IRMv1 [Arj+19], a problem setting to solve IRM, to clarify the relationship between the problem formulation and optimizer selection in the OOD problem.
Revised: We investigate both ERM and IRMv1 [Arj+19], a problem setting to solve IRM, to clarify the relationship between the problem formulation and optimizer selection in the OOD problem.

Original: For all datasets except for the Backgrounds Challenge dataset, we compare five optimizers.
Revised: For all datasets except for the Backgrounds Challenge dataset, we compare five optimizers.

Original: For the Backgrounds Challenge dataset, we compare only Momentum SGD and Adam due to the computational cost.
Revised: For the Backgrounds Challenge dataset, we compare only Momentum SGD and Adam due to the computational cost.

Original: We describe the configurations of hyperparameters and protocol for the experiments in further detail in Appendix F and Appendix E respectively.
Revised: We describe the configurations of hyperparameters and protocol for the experiments in further detail in Appendix F and Appendix E respectively.

Original: The remaining experimental settings of environment are explained in Appendix D.
Revised: The remaining experimental settings of environment are explained in Appendix D.

Original: 3.2 Experimental Results
Revised: 3.2 Experimental Results

Original: Figure 1 shows the relationship between the in- [Figure] d i [Figure]
Revised: Figure 1 shows the relationship between the in-distribution accuracy and the OOD accuracy in the

Original: t r i [Figure]
Revised: ERM setting.

Original: u t i o n
Revised: The x-axis of the plot is the in-distribution accuracy and the y-axis is the OOD accuracy.

Original: [Figure]
Revised: To make the trend more clear, the in-distribution accuracy corresponding to the x-axis is divided into 10 bins, and the average performance of the OOD accuracy in each bin is shown on the y-axis.

Original: We compared Momentum SGD as the best non-adaptive optimizer, with Adam as the best adaptive optimizer.
Revised: We compared Momentum SGD as the best non-adaptive optimizer, with Adam as the best adaptive optimizer.

Original: In our area of interest, where a high in-distribution performance is achieved, such as top 10% trial, Momentum SGD outperforms Adam on 8 of the 10 datasets.
Revised: In our area of interest, where a high in-distribution performance is achieved, such as top 10% trial, Momentum SGD outperforms Adam on 8 of the 10 datasets.

Original: This indicates that non adaptive optimizer is more advantageous than adaptive optimizer in OOD, even though the performance is similar in the IID environment.
Revised: This indicates that non adaptive optimizer is more advantageous than adaptive optimizer in OOD, even though the performance is similar in the IID environment.

Original: A detailed explanation of the experimental results for each dataset is given in Appendix G
Revised: A detailed explanation of the experimental results for each dataset is given in Appendix G.

Original: 3.3 Correlation Behaviors
Revised: 3.3 Correlation Behaviors

Original: Our results show that three typical types of behavior are observed in terms of the correlation between in-distribution performance and OOD performance for different datasets.
Revised: Our results show that three typical types of behavior are observed in terms of the correlation between in-distribution performance and OOD performance for different datasets.

Original: Detailed results of the experiments on all datasets are shown in Appendix H.3.
Revised: Detailed results of the experiments on all datasets are shown in Appendix H.3.

Original: We follow Miller et al.
Revised: We follow Miller et al.

Original: [Mil+21] who used a probit transform to show relationship.
Revised: [Mil+21] who used a probit transform to show relationship.

Original: The three types are increasing return, linear return, and diminishing return.
Revised: The three types are increasing return, linear return, and diminishing return.

Original: These show how much performance in OOD can be expected if we increase the in-distribution performance.
Revised: These show how much performance in OOD can be expected if we increase the in-distribution performance.

Original: We show in Appendix H.3 the results of our experiments without a probit transform.
Revised: We show in Appendix H.3 the results of our experiments without a probit transform.

Original: These results confirm that diminishing returns, in particular, are not necessarily linear even before probit transformation.
Revised: These results confirm that diminishing returns, in particular, are not necessarily linear even before probit transformation.

Original: This is consistent with the recent point by [Wen+22; Ten+22] that the accuracy of IID and
Revised: This is consistent with the recent point by [Wen+22; Ten+22] that the accuracy of IID and

Original: OOD differ depending on the dataset.
Revised: OOD differ depending on the dataset.

Original: Furthermore, as recent work, [Bae+22] states, whether or not the linear return actually occurs depends on the problem set.
Revised: Furthermore, as recent work, [Bae+22] states, whether or not the linear return actually occurs depends on the problem set.

Original: Our results support their claims, and we have comprehensively addressed data sets and problem sets that Miller et al.
Revised: Our results support their claims, and we have comprehensively addressed data sets and problem sets that Miller et al.

Original: [Mil+21] did not address and uncovered trends that they did not reveal.
Revised: [Mil+21] did not address and uncovered trends that they did not reveal.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: We conduct an exhaustive empirical comparison of the generalization performance of various optimizers under different practical distributional shifts.
Revised: We conduct an exhaustive empirical comparison of the generalization performance of various optimizers under different practical distributional shifts.

Original: The investigation elucidates how optimizer selection affects OOD generalization.
Revised: The investigation elucidates how optimizer selection affects OOD generalization.

Original: As the main claim, the answer to our research objective is that the non-adaptive optimizer is superior to the adaptive optimizer in terms of OOD generalization.
Revised: As the main claim, the answer to our research objective is that the non-adaptive optimizer is superior to the adaptive optimizer in terms of OOD generalization.

Original: This is supported by following evidence: i) when comparing in-distribution accuracy is the same,
Revised: This is supported by following evidence: i) when comparing in-distribution accuracy is the same,

Original: OOD accuracy of non-adaptive optimizers is greater than that of adaptive optimizers (Figure 1); ii) on average and top performance, the OOD accuracy of non-adaptive optimizer is higher (Figures 3, 4, and 5).
Revised: OOD accuracy of non-adaptive optimizers is greater than that of adaptive optimizers (Figure 1); ii) on average and top performance, the OOD accuracy of non-adaptive optimizer is higher (Figures 3, 4, and 5).

Original: All these points support our main claim that non-adaptive optimizers are superior in OOD generalization within our exhaustive experiments.
Revised: All these points support our main claim that non-adaptive optimizers are superior in OOD generalization within our exhaustive experiments.

Original: We have tuned Adam to the range that it is used in practice and have updated Adam’s scores on all OOD datasets which use Adam optimizer as default for benchmarks.
Revised: We have tuned Adam to the range that it is used in practice and have updated Adam’s scores on all OOD datasets which use Adam optimizer as default for benchmarks.

Original: This supports the soundness of our experiments.
Revised: This supports the soundness of our experiments.

Original: Overall, we can conclude that optimizer selection influences OOD generalization in the cases we are interested in.
Revised: Overall, we can conclude that optimizer selection influences OOD generalization in the cases we are interested in.

Original: Future research should consider the algorithm or loss function and optimizers in the
Revised: Future research should consider the algorithm or loss function and optimizers in the

Original: OOD problem.
Revised: OOD problem.

Original: The results of IRM show a trend similar to ERM’s, but a more detailed analysis is needed to consider the differences in loss landscapes.
Revised: The results of IRM show a trend similar to ERM’s, but a more detailed analysis is needed to consider the differences in loss landscapes.

Original: All these points support our main claim that non-adaptive optimizers are superior in OOD generalization within our exhaustive experiments.
Revised: All these points support our main claim that non-adaptive optimizers are superior in OOD generalization within our exhaustive experiments.

Original: Finally, we would like to mention the limitations of our work.
Revised: Finally, we would like to mention the limitations of our work.

Original: One limitation is that we did not study recently proposed and less popular optimizers.
Revised: One limitation is that we did not study recently proposed and less popular optimizers.

Original: The choice of optimizers we study is in line with previous work [Cho+19], [SSH21] on optimizer comparison and with most recent OOD work; those studies overwhelmingly focused on Adam rather than e.g. AdamW
Revised: The choice of optimizers we study is in line with previous work [Cho+19], [SSH21] on optimizer comparison and with most recent OOD work; those studies overwhelmingly focused on Adam rather than e.g. AdamW

Original: [LH17].
Revised: [LH17].

Original: Similarly other less popular optimizers such as SWA [Izm+18], SWAD [Cha+21], SAM [For+20], and second-order methods, have been omitted to allow for a more extensive study of the chosen methods.
Revised: Similarly other less popular optimizers such as SWA [Izm+18], SWAD [Cha+21], SAM [For+20], and second-order methods, have been omitted to allow for a more extensive study of the chosen methods.

Original: Another limitation is that we employed a total of four models used in the DomainBed (ConvNet and
Revised: Another limitation is that we employed a total of four models used in the DomainBed (ConvNet and

Original: ResNet50), Backgrounds Challenge (ResNet50), and WILDS (DistilBERT [San+19]) benchmarks, but we did not evaluated other deep neural network architectures such as the Vision Transformer [Dos+20].
Revised: ResNet50), Backgrounds Challenge (ResNet50), and WILDS (DistilBERT [San+19]) benchmarks, but we did not evaluated other deep neural network architectures such as the Vision Transformer [Dos+20]

Original: Acknowledgments
Revised: Acknowledgments

Original: We would like to thank Kilian Fatras, Divyat Mahajan and Mehrnaz Mofakhami for their helpful feedback.
Revised: We would like to thank Kilian Fatras, Divyat Mahajan and Mehrnaz Mofakhami for their helpful feedback.

Original: This work was supported by the Masason Foundation Fellowship awarded to Hiroki
Revised: This work was supported by the Masason Foundation Fellowship awarded to Hiroki

Original: Naganuma.
Revised: Naganuma.

Original: Computational resource of AI Bridging Cloud Infrastructure (ABCI) was awarded by "ABCI Grand Challenge" Program, National Institute of Advanced Industrial Science and Technology (AIST), and resource of the TSUBAME 3.0 was awarded by the TSUBAME Grand Challenge
Revised: Computational resource of AI Bridging Cloud Infrastructure (ABCI) was awarded by "ABCI Grand Challenge" Program, National Institute of Advanced Industrial Science and Technology (AIST), and resource of the TSUBAME 3.0 was awarded by the TSUBAME Grand Challenge

Original: Program, Tokyo Institute of Technology.
Revised: Program, Tokyo Institute of Technology.

Original: C Preliminaries
Revised: C Preliminaries

Original: C.1 Optimizers Subjected to Our Analysis . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.1 Optimizers Subjected to Our Analysis . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.1.
Revised: C.1.

Original: SGD . . .
Revised: SGD . . .

Original: C.1.
Revised: C [Table]

Original: Momentum-SGD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Momentum-SGD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.1.
Revised: C [Table]

Original: Nesterov Momentum . .
Revised: Nesterov Momentum . .

Original: C.1.
Revised: C [Table]

Original: RMSprop . . .
Revised: RMSprop . . .

Original: C.1.
Revised: C [Table]

Original: Adam . . .
Revised: Adam . . .

Original: C.2 Out-of-Distribution Generalization Datasets . .
Revised: C.2 Out-of-Distribution Generalization Datasets . .

Original: C.2.
Revised: C.2.

Original: DomainBed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: DomainBed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.2.
Revised: C [Table]

Original: Backgrounds Challenge Dataset . . . . . . . . . . . . . . . . . . . . . . .
Revised: Backgrounds Challenge Dataset . . . . . . . . . . . . . . . . . . . . . . .

Original: C.2.
Revised: C [Table]

Original: Amazon-WILDS and CivilComments-WILDS Dataset . . . . . . . . . . .
Revised: Amazon-WILDS and CivilComments-WILDS Dataset . . . . . . . . . . .

Original: D Implementation and Environment for Experiments
Revised: D Implementation and Environment for Experiments

Original: E
Revised: E

Original: Experimental Protocol
Revised: Experimental Protocol

Original: E.1 DomainBed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.1 DomainBed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.2 Backgrounds Challenge Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.2 Backgrounds Challenge Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.3 WILDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.3 WILDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F Hyperparameters and Detailed Configurations
Revised: F Hyperparameters and Detailed Configurations

Original: F.
Revised: F.

Original: DomainBed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: DomainBed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.
Revised: F.

Original: Backgrounds Challenge Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Backgrounds Challenge Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.
Revised: F.

Original: WILDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: WILDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G Main Results
Revised: G Main Results

Original: G.1 Experimental Results on DomainBed . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.1 Experimental Results on DomainBed . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2 Experimental Results on Backgrounds Challenge . . . . . . . . . . . . . . . . . .
Revised: G.2 Experimental Results on Backgrounds Challenge . . . . . . . . . . . . . . . . . .

Original: G.3 Experimental Results on WILDS . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.3 Experimental Results on WILDS . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.4 Experimental Results: Correlation Behaviors . . . . . . . . . . . . . . . . . . . .
Revised: G.4 Experimental Results: Correlation Behaviors . . . . . . . . . . . . . . . . . . . .

Original: H Full Results of Experiments
Revised: H Full Results of Experiments

Original: H.1 Full Results of Boxplot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: H.1 Full Results of Boxplot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H.1.

Revised: H.1.

Original: Full Results of Filtered Boxplot . . . . . . . . . . . . . . . . . . . . . . .
Revised: Full Results of Filtered Boxplot . . . . . . . . . . . . . . . . . . . . . . .

Original: H.3.1 ERM . . . . . . . . . . . . . . . . .{+H.1.
 Full Results of H.2 Full Results of Like Plots+} . . . . . . . . . . . . . . . . . . . .
Revised: {+H.1.
 Full Results of H.2 Full Results of Like Plots+} . . . . . . . . . . . . . . . . . . . .

Revised: H.3 Full Results

Revised: Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: H.3.1 ERM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H.3.2 IRM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: H.3.2 IRM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: I Additional Results of Experiments
Revised: I Additional Results of Experiments

Original: I.
Revised: I.

Original: Probit Transformed Scatter Plot . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Probit Transformed Scatter Plot . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: I.1.
Revised: I.1.

Original: ERM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: ERM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: I.1.
Revised: I.1.

Original: IRM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: IRM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: I.
Revised: I.

Original: Results of OOD Accuracy when  is the trial budget . . . . . . . . .
Revised: Results of OOD Accuracy when horizontal axis is the trial budget . . . . . . . . .

Original: I.
Revised: I.

Original: Learning Curve of ColoredMNIST . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Learning Curve of ColoredMNIST . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: I.
Revised: I.

Original: Early Stopping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Early Stopping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: J Histgram of Hyperparameters
Revised: J Soundness Check of Our Experiments

Original: K The Number of Trials
Revised: J. Histgram of Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: . Best OOD Performance Comparison against with Existing Benckmark

Original: A Appendix B Related Works B.1 Optimizer Selection
Revised: A Appendix B Related Works B.1 Optimizer Selection

Original: Understanding the characteristics of the many optimizers proposed for deep neural network training [SSH21] is of great importance to the machine learning research community.
Revised: Understanding the characteristics of the many optimizers proposed for deep neural network training [SSH21] is of great importance to the machine learning research community.

Original: In terms of convergence, preconditioned optimizers, including Adam, are known to be superior to non-adaptive optimizers [KB15; Liu+19; Ama98].
Revised: In terms of convergence, preconditioned optimizers, including Adam, are known to be superior to non-adaptive optimizers [KB15; Liu+19; Ama98].

Original: While preconditioned optimization methods seem to be better than non-pre-conditioned ones in terms of convergence, Zhang et al.
Revised: While preconditioned optimization methods seem to be better than non-pre-conditioned ones in terms of convergence, Zhang et al.

Original: argued that there is a trade-off between generalization performance and convergence rate [Zha+19].
Revised: argued that there is a trade-off between generalization performance and convergence rate [Zha+19].

Original: Wadia et al.
Revised: Wadia et al.

Original: also reported that the preconditioned methods, especially second-order methods, do not provide high generalization performance either empirically or theoretically [Wad+20].
Revised: also reported that the preconditioned methods, especially second-order methods, do not provide high generalization performance either empirically or theoretically [Wad+20].

Original: With a simple theoretical and exhaustive empirical analysis, Wilson et al. showed that adaptive optimizers are worse at generalization than simple SGD
Revised: With a simple theoretical and exhaustive empirical analysis, Wilson et al. showed that adaptive optimizers are worse at generalization than simple SGD

Original: [Wil+17].
Revised: [Wil+17].

Original: Balles and Hennig also reported that Adam generalizes worse than Momentum SGD
Revised: Balles and Hennig also reported that Adam generalizes worse than Momentum SGD

Original: [BH18].
Revised: [BH18].

Original: Contrary to these studies, Schneider et al.
Revised: Contrary to these studies, Schneider et al.

Original: found that no single optimizer is the “best" in general [SBH19].
Revised: found that no single optimizer is the “best" in general [SBH19].

Original: Similarly, Schmidt et al. claimed that the optimizer performance varies from task to task [SSH20].
Revised: Similarly, Schmidt et al. claimed that the optimizer performance varies from task to task [SSH20].

Original: Choi et al.
Revised: Choi et al.

Original: have come to a different conclusion from all the studies cited above [Cho+19].
Revised: have come to a different conclusion from all the studies cited above [Cho+19].

Original: Choi et al.
Revised: Choi et al.

Original: tuned not only the learning rate but also other hyperparameters of adaptive methods.
Revised: tuned not only the learning rate but also other hyperparameters of adaptive methods.

Original: As a result, they found that well-tuned adaptive optimizers never underperform simple gradient methods.
Revised: As a result, they found that well-tuned adaptive optimizers never underperform simple gradient methods.

Original: The study of Choi et al.
Revised: The study of Choi et al.

Original: is a solid report in that it is the most exhaustive hyperparameter tuning of any study to date.
Revised: is a solid report in that it is the most exhaustive hyperparameter tuning of any study to date.

Original: These studies provided insights on how optimizer selection influences generalization.
Revised: These studies provided insights on how optimizer selection influences generalization.

Original: However, the focus of these studies was on the classical supervised learning setting, in which the test distribution is assumed to be the same as the training one.
Revised: However, the focus of these studies was on the classical supervised learning setting, in which the test distribution is assumed to be the same as the training one.

Original: Our research differs from them in that we investigate the influence of optimizer selection on the OOD generalization.
Revised: Our research differs from them in that we investigate the influence of optimizer selection on the OOD generalization.

Original: B.2 Out-of-Distribution Generalization Problem
Revised: B.2 Out-of-Distribution Generalization Problem

Original: Taming the distributional shift is a big challenge in machine learning research [SK12; Ben+10;
Revised: Taming the distributional shift is a big challenge in machine learning research [SK12; Ben+10;

Original: PY09; Sze+14; Arj+19].
Revised: PY09; Sze+14; Arj+19].

Original: Geirhos et al.
Revised: Geirhos et al.

Original: argued that many modern deep neural network models sometimes learn shortcut features instead of intended features and overfit to a specific dataset [Gei+20].
Revised: argued that many modern deep neural network models sometimes learn shortcut features instead of intended features and overfit to a specific dataset [Gei+20].

Original: These models show high in-distribution performance, making predictions using shortcut features but performing poorly in OOD environments.
Revised: These models show high in-distribution performance, making predictions using shortcut features but performing poorly in OOD environments.

Original: Here, the choice of hyperparameters and optimizers is an inductive bias in learning, leading to convergence to a different local minimum.
Revised: Here, the choice of hyperparameters and optimizers is an inductive bias in learning, leading to convergence to a different local minimum.

Original: Some studies have focused on generalization with adversarial noise to evaluate the impact of optimizer selection on OOD generalization.
Revised: Some studies have focused on generalization with adversarial noise to evaluate the impact of optimizer selection on OOD generalization.

Original: For instance, the theoretical and empirical analysis by Khoury showed that SGD is more robust against adversarial noise than the adaptive optimizers [Kho19].
Revised: For instance, the theoretical and empirical analysis by Khoury showed that SGD is more robust against adversarial noise than the adaptive optimizers [Kho19].

Original: Wang et al.
Revised: Wang et al.

Original: argued that adversarial examples to some methods are not necessarily adversarial to others [Wan+19].
Revised: argued that adversarial examples to some methods are not necessarily adversarial to others [Wan+19].

Original: Other related works also pertain to OOD problems.
Revised: Other related works also pertain to OOD problems.

Original: For example, Abdelzad et al. found that the best optimizers for OOD detection vary by experimental setting [ACS20].
Revised: For example, Abdelzad et al. found that the best optimizers for OOD detection vary by experimental setting [ACS20].

Original: Metz et al.
Revised: Metz et al.

Original: reported that a learned optimizer somehow unexpectedly outperformed a human-designed optimizer in terms of the OOD generalization [Met+20].
Revised: reported that a learned optimizer somehow unexpectedly outperformed a human-designed optimizer in terms of the OOD generalization [Met+20].

Original: These studies provide valuable insights on optimizer selection for the OOD problem.
Revised: These studies provide valuable insights on optimizer selection for the OOD problem.

Original: However, we emphasize that the previous works discussed above explored hyperparameters in a limited range.
Revised: However, we emphasize that the previous works discussed above explored hyperparameters in a limited range.

Original: Khoury conducted the most exhaustive hyperparameter search but searched only the learning rate for adaptive optimizers [Kho19].
Revised: Khoury conducted the most exhaustive hyperparameter search but searched only the learning rate for adaptive optimizers [Kho19].

Original: As we briefly explain in Section 1, the exhaustiveness of the hyperparameter search is crucial for empirical investigation of an optimizer’s effect.
Revised: As we briefly explain in Section 1, the exhaustiveness of the hyperparameter search is crucial for empirical investigation of an optimizer’s effect.

Original: Thus, we basically follow [Cho+19], which most exhaustively searched hyperparameters for optimizer comparison and explored more hyperparameters than did previous studies.
Revised: Thus, we basically follow [Cho+19], which most exhaustively searched hyperparameters for optimizer comparison and explored more hyperparameters than did previous studies.

Original: Here we emphasize that only shifts such as adversarial noise have been studied in previous studies of optimization selection for OOD generalization.
Revised: Here we emphasize that only shifts such as adversarial noise have been studied in previous studies of optimization selection for OOD generalization.

Original: Thus, we use a much more diverse set of real OOD datasets, including image classification and NLP tasks where the distributional shift is significant, covariate shift, correlation shift, subpopulation shift, and background shift, not only domain generalization.
Revised: Thus, we use a much more diverse set of real OOD datasets, including image classification and NLP tasks where the distributional shift is significant, covariate shift, correlation shift, subpopulation shift, and background shift, not only domain generalization.

Original: The set of datasets we explored is the most exhaustive for evaluating the optimizer’s role in OOD generalization, as far as we know.
Revised: The set of datasets we explored is the most exhaustive for evaluating the optimizer’s role in OOD generalization, as far as we know.

Original: C Preliminaries C.1 Optimizers Subjected to Our Analysis
Revised: C Preliminaries C.1 Optimizers Subjected to Our Analysis

Original: Similar to previous studies [Wil+17; SBH19; Cho+19], we compare two types of optimizers.
Revised: Similar to previous studies [Wil+17; SBH19; Cho+19], we compare two types of optimizers.

Original: The first one is non-adaptive optimizers.
Revised: The first one is non-adaptive optimizers.

Original: The update equation at iteration t of model parameter θ t is as follows:
Revised: The update equation at iteration t of model parameter θ t is as follows:

Original: [Equation]
Revised: [Equation]

Original: where η t is learning rate, ℓ ( θ ) is the loss, and ˜ ∇ θ t − 1 is the stochastic gradient, in the particular case of stochastic gradient descent, γ = 0 .
Revised: where η t is learning rate, ℓ ( θ ) is the loss, and ˜ ∇ θ t − 1 is the stochastic gradient, in the particular case of stochastic gradient descent, γ = 0 .

Original: Optimizers with momentum terms such as Momentum SGD [Pol64], and Nesterov momentum [Nes03] are also classified as non-adaptive optimizers, and γ is the parameter for controlling the momentum term.
Revised: Optimizers with momentum terms such as Momentum SGD [Pol64], and Nesterov momentum [Nes03] are also classified as non-adaptive optimizers, and γ is the parameter for controlling the momentum term.

Original: For Nesterov momentum, ℓ ( θ t − 1 ) should be replaced ℓ ( θ t − 1 − γ v t − 1 ) .
Revised: For Nesterov momentum, ℓ ( θ t − 1 ) should be replaced ℓ ( θ t − 1 − γ v t − 1 ) .

Original: The second type of optimizer is adaptive methods.
Revised: The second type of optimizer is adaptive methods.

Original: Adam and RMSprop are adaptive optimizers and they can be written in the form of the generic adaptive optimization method.
Revised: Adam and RMSprop are adaptive optimizers and they can be written in the form of the generic adaptive optimization method.

Original: The generic adaptive optimization method can be written as in
Revised: The generic adaptive optimization method can be written as in

Original: Algorithm 1.
Revised: Algorithm 1.

Original: This is based on what [Liu+20] and [RKK18] propose.
Revised: This is based on what [Liu+20] and [RKK18] propose.

Original: Require calculate
Revised: Require calculate

Original: : for
Revised: : for

Original: ← θ t − 1 − η t w t l t (Update parameters) 6: end for
Revised: ← θ t − 1 − η t w t l t (Update parameters) 6: end for

Original: The choice of optimizers we study is in line with previous work on optimizer comparison [Cho+19] and with most recent OOD work; those studies overwhelmingly focused on Adam rather than e.g.
Revised: The choice of optimizers we study is in line with previous work on optimizer comparison [Cho+19] and with most recent OOD work; those studies overwhelmingly focused on Adam rather than e.g.

Original: AdamW.
Revised: AdamW.

Original: It is this overwhelming popularity that informed our study design: studying Adam yields more relevant results for current practice.
Revised: It is this overwhelming popularity that informed our study design: studying Adam yields more relevant results for current practice.

Original: A concrete formulation of these five optimizer update formulas is as follows.
Revised: A concrete formulation of these five optimizer update formulas is as follows.

Original: C.1.1 SGD
Revised: C.1.1 SGD

Original: [Equation]
Revised: [Equation]

Original: C.1.2 Momentum-
Revised: C.1.2 Momentum-

Original: [Equation]
Revised: [Equation]

Original: C.1.3 Nesterov Momentum
Revised: C.1.3 Nesterov Momentum

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: C.1.5 Adam
Revised: C.1.5 Adam

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: C.2 Out-of-Distribution Generalization Datasets
Revised: C.2 Out-of-Distribution Generalization Datasets

Original: Image Classification Datasets: DomainBed consists of a set of benchmark datasets for domain generalization, which includes Colored MNIST [Arj+19], Rotated MNIST [Ghi+15], PACS [FXR13],
Revised: Image Classification Datasets: DomainBed consists of a set of benchmark datasets for domain generalization, which includes Colored MNIST [Arj+19], Rotated MNIST [Ghi+15], PACS [FXR13],

Original: VLCS [Li+17], Office-Home [Ven+17], Terra Incognita [BVP18], and DomainNet [Pen+19].
Revised: VLCS [Li+17], Office-Home [Ven+17], Terra Incognita [BVP18], and DomainNet [Pen+19].

Original: These datasets contain a variety of distributional shifts.
Revised: These datasets contain a variety of distributional shifts.

Original: VLCS is a set of different image datasets, for example, images of birds from several datasets.
Revised: VLCS is a set of different image datasets, for example, images of birds from several datasets.

Original: Terra Incognita is a dataset consisting of images taken by cameras at different locations.
Revised: Terra Incognita is a dataset consisting of images taken by cameras at different locations.

Original: The difference in the location of the camera corresponds to the difference in the domain.
Revised: The difference in the location of the camera corresponds to the difference in the domain.

Original: PACS, Office-Home, and DomainNet are image datasets whose style varies by domain.
Revised: PACS, Office-Home, and DomainNet are image datasets whose style varies by domain.

Original: For example, an image of PACS in one domain is photography, while that in another domain is a sketch.
Revised: For example, an image of PACS in one domain is photography, while that in another domain is a sketch.

Original: Rotated MNIST is an artificially generated dataset of domains that have been given different rotation angles.
Revised: Rotated MNIST is an artificially generated dataset of domains that have been given different rotation angles.

Original: Colored MNIST is an anomaly in the DomainBed dataset.
Revised: Colored MNIST is an anomaly in the DomainBed dataset.

Original: P ( Y | X ) remains the same for all datasets (covariate shift) except in Colored MNIST.
Revised: P ( Y | X ) remains the same for all datasets (covariate shift) except in Colored MNIST.

Original: This dataset is designed to make models fail by exploiting spurious correlations in training environments.
Revised: This dataset is designed to make models fail by exploiting spurious correlations in training environments.

Original: In particular, the dataset is such that the model can only exploit the source of spurious correlation (color) and achieve a very high training accuracy without relying on the true invariant source of correlation (shape).
Revised: In particular, the dataset is such that the model can only exploit the source of spurious correlation (color) and achieve a very high training accuracy without relying on the true invariant source of correlation (shape).

Original: In contrast, none of the other datasets have such strong spuriousness.
Revised: In contrast, none of the other datasets have such strong spuriousness.

Original: The strength of the spurious correlation flips in the test domain and thus it induces a negative correlation between the validation and the test accuracy.
Revised: The strength of the spurious correlation flips in the test domain and thus it induces a negative correlation between the validation and the test accuracy.

Original: The Backgrounds Challenge dataset measures a model’s robustness against background shift [Xia+21].
Revised: The Backgrounds Challenge dataset measures a model’s robustness against background shift [Xia+21].

Original: A model is trained on an image and evaluated on the same image with a different background.
Revised: A model is trained on an image and evaluated on the same image with a different background.

Original: If the model exploits the background features during training, it will be fooled during evaluation.
Revised: If the model exploits the background features during training, it will be fooled during evaluation.

Original: Therefore, if the model strongly depends on the background, this dataset is a difficult OOD dataset, while if the model does not, this dataset is easy to model.
Revised: Therefore, if the model strongly depends on the background, this dataset is a difficult OOD dataset, while if the model does not, this dataset is easy to model.

Original: Natural Language Processing (NLP) Datasets: The CivilComments-WILDS dataset is cast as a subpopulation shift problem.
Revised: Natural Language Processing (NLP) Datasets: The CivilComments-WILDS dataset is cast as a subpopulation shift problem.

Original: The shift problem tackles a binary classification problem that predicts the toxicity of comments on articles, and domain vectors are assigned according to whether the comment refers to one of eight demographic identities.
Revised: The shift problem tackles a binary classification problem that predicts the toxicity of comments on articles, and domain vectors are assigned according to whether the comment refers to one of eight demographic identities.

Original: In the subpopulation shift, the test distribution is a subpopulation of the training distribution.
Revised: In the subpopulation shift, the test distribution is a subpopulation of the training distribution.

Original: The Amazon-WILDS dataset has the characteristics of a hybrid shift of a subpopulation shift and domain shift.
Revised: The Amazon-WILDS dataset has the characteristics of a hybrid shift of a subpopulation shift and domain shift.

Original: This dataset is cast as the problem of estimating a rating of 1–5 from the rating comments of each user.
Revised: This dataset is cast as the problem of estimating a rating of 1–5 from the rating comments of each user.

Original: In this kind of dataset, each user corresponds to a domain, and the goal is to produce a high performance for all user comments.
Revised: In this kind of dataset, each user corresponds to a domain, and the goal is to produce a high performance for all user comments.

Original: C.2.1 DomainBed
Revised: C.2.1 DomainBed

Original: DomainBed consists of sub-datasets shown in Table 2, where we exclude Terra Incognita and Rotated
Revised: DomainBed consists of sub-datasets shown in Table 2, where we exclude Terra Incognita and Rotated

Original: MNIST as stated in Section 2.1.
Revised: MNIST as stated in Section 2.1.

Original: We summarize the dataset information in the Table by referring to [GL21].
Revised: We summarize the dataset information in the Table by referring to [GL21].

Original: [Table]
Revised: [Table]

Original: Colored MNIST is a dataset for binary classification of MNIST dataset [Arj+19].
Revised: Colored MNIST is a dataset for binary classification of MNIST dataset [Arj+19].

Original: The digits from 0 to 4 are labeled 0, and those greater than 5 are labeled 1, and the task is to classify these classes successfully.
Revised: The digits from 0 to 4 are labeled 0, and those greater than 5 are labeled 1, and the task is to classify these classes successfully.

Original: However, each digit is also colored by either red or green.
Revised: However, each digit is also colored by either red or green.

Original: This is for having models to confuse the important feature for classification.
Revised: This is for having models to confuse the important feature for classification.

Original: The domain d indicates the correlation of the label with color.
Revised: The domain d indicates the correlation of the label with color.

Original: For example, if the domain is 0.1, the correlation between, say red, with the number smaller than 5 is 0.1.
Revised: For example, if the domain is 0.1, the correlation between, say red, with the number smaller than 5 is 0.1.

Original: Furthermore, the label is flipped at a constant rate: in this paper, 15 % label is flipped.
Revised: Furthermore, the label is flipped at a constant rate: in this paper, 15 % label is flipped.

Original: Therefore, the correlation between color and digit is d , while between label and digits is 0.85.
Revised: Therefore, the correlation between color and digit is d , while between label and digits is 0.85.

Original: That is, what models should learn is the correlation between label and noise, resulting in classification accuracy of 0.85.
Revised: That is, what models should learn is the correlation between label and noise, resulting in classification accuracy of 0.85.

Original: However, if the model exploits spurious correlation of the domain, it will learn the correlation between label and color, resulting in training accuracy being 0.9 but test accuracy being 0.1 in this case.
Revised: However, if the model exploits spurious correlation of the domain, it will learn the correlation between label and color, resulting in training accuracy being 0.9 but test accuracy being 0.1 in this case.

Original: PACS and Office-Home are image datasets whose domain determines the style of the image.
Revised: PACS and Office-Home are image datasets whose domain determines the style of the image.

Original: These are benchmark datasets for domain generalization.
Revised: These are benchmark datasets for domain generalization.

Original: VLCS is a set of different photographic datasets,
Revised: VLCS is a set of different photographic datasets,

Original: PASCAL VOC [Eve+10], LabelMe [Rus+08],
Revised: PASCAL VOC [Eve+10], LabelMe [Rus+08],

Original: Caltech101 [FFP04], and SUN09 [Cho+10].
Revised: Caltech101 [FFP04], and SUN09 [Cho+10].

Original: PASCAL VOC, LabelMe, and SUN09 are benchmark datasets for object detection.
Revised: PASCAL VOC, LabelMe, and SUN09 are benchmark datasets for object detection.

Original: Caltech101 is 101 classes image datasets, where each class has 40samples.
Revised: Caltech101 is 101 classes image datasets, where each class has 40samples.

Original: DomainNet is a large dataset proposed for the study of domain generalization.
Revised: DomainNet is a large dataset proposed for the study of domain generalization.

Original: The number of classes, domains, and dataset size is the largest in the DomainBed dataset.
Revised: The number of classes, domains, and dataset size is the largest in the DomainBed dataset.

Original: TerraIncognita is a dataset consisting of images taken by cameras at different locations.
Revised: TerraIncognita is a dataset consisting of images taken by cameras at different locations.

Original: The difference in the camera’s location corresponds to the difference in the domain.
Revised: The difference in the camera’s location corresponds to the difference in the domain.

Original: RotatedMNIST is a dataset that artificially rotates MNIST and divides the domain according to the rotation angle.
Revised: RotatedMNIST is a dataset that artificially rotates MNIST and divides the domain according to the rotation angle.

Original: The number in the domain corresponds to the rotation angle.
Revised: The number in the domain corresponds to the rotation angle.

Original: We leave one domain for the final evaluation and use the remaining domains for training.
Revised: We leave one domain for the final evaluation and use the remaining domains for training.

Original: To evaluate the performance during training, we split the data of each domain into training data and validation data.
Revised: To evaluate the performance during training, we split the data of each domain into training data and validation data.

Original: The split ratio is 80 % for training and 20 % for validation.
Revised: The split ratio is 80 % for training and 20 % for validation.

Original: We take an average of test accuracies and validation accuracies across domains, respectively, and use them to evaluate the OOD generalization.
Revised: We take an average of test accuracies and validation accuracies across domains, respectively, and use them to evaluate the OOD generalization.

Original: C.2.2 Backgrounds Challenge Dataset
Revised: C.2.2 Backgrounds Challenge Dataset

Original: In Section E.2, we explained that we use the subset of ImageNet (ORIGINAL).
Revised: In Section E.2, we explained that we use the subset of ImageNet (ORIGINAL).

Original: ORIGINAL consists of nine classes displayed in table 3.
Revised: ORIGINAL consists of nine classes displayed in table 3.

Original: These classes are synthetically created from ImageNet classes based on WordNet ID.
Revised: These classes are synthetically created from ImageNet classes based on WordNet ID.

Original: This table is a copy of a table in the original paper [Xia+21].
Revised: This table is a copy of a table in the original paper [Xia+21].

Original: This dataset is filtered to balance samples across classes.
Revised: This dataset is filtered to balance samples across classes.

Original: We follow the same filtering procedure as the original paper.
Revised: We follow the same filtering procedure as the original paper.

Original: For further details, please refer to the original paper [Xia+21].
Revised: For further details, please refer to the original paper [Xia+21].

Original: C.2.3 Amazon-WILDS and CivilComments-WILDS Dataset
Revised: C.2.3 Amazon-WILDS and CivilComments-WILDS Dataset

Original: WILDS is a set ob benchmark datasets with distributional shift and their variants: iWildCam [BCG20], Camelyon17 [Ban+18], RxRx

Revised: WILDS is a set ob benchmark datasets with distributional shift and their variants: iWildCam [BCG20], Camelyon17 [Ban+18], RxRx

Original: [Tay+19], OGB-MolPCBA [Hu+20], GlobalWheat [Dav+20; Dav+21], CivilComments [Bor+19], FMoW [Chr+18], PovertyMap [Yeh+20], Amazon [NLM19], and Py150 [Lu+21; RBV16].
Revised: [Tay+19], OGB-MolPCBA [Hu+20], GlobalWheat [Dav+20; Dav+21], CivilComments [Bor+19], FMoW [Chr+18], PovertyMap [Yeh+20], Amazon [NLM19], and Py150 [Lu+21; RBV16].

Original: We use CivilComments
Revised: We use CivilComments

Original: [Table]
Revised: [Table]

Original: D Implementation and Environment for Experiments
Revised: D Implementation and Environment for Experiments

Original: We perform our experiment with ABCI (AI Bridging Cloud Infrastructure), a supercomputer owned by the National Institute of Advanced Industrial Science and Technology, and TSUBAME, a supercomputer owned by the Tokyo Institute of Technology.
Revised: We perform our experiment with ABCI (AI Bridging Cloud Infrastructure), a supercomputer owned by the National Institute of Advanced Industrial Science and Technology, and TSUBAME, a supercomputer owned by the Tokyo Institute of Technology.

Original: The total amount of computation is 90,
Revised: The total amount of computation is 90,

Original: GPU hours.
Revised: GPU hours.

Original: All codes for experiments are modifications of the codes provided by the authors who introduced the datasets [GL21; Koh+21; Xia+21].
Revised: All codes for experiments are modifications of the codes provided by the authors who introduced the datasets [GL21; Koh+21; Xia+21].

Original: Licenses of the codes are MIT license for DomainBed [GL21] and WILDS [Koh+21].
Revised: Licenses of the codes are MIT license for DomainBed [GL21] and WILDS [Koh+21].

Original: The code of Backgrounds Challenge does not indicate the license.
Revised: The code of Backgrounds Challenge does not indicate the license.

Original: Our code can be found at the link below.
Revised: Our code can be found at the link below.

Original: https://github.com/Hiroki11x/Optimizer_Comparison_OOD
Revised: https://github.com/Hiroki11x/Optimizer_Comparison_OOD

Original: E Experimental Protocol Hyperparameter Tuning:
Revised: E Experimental Protocol Hyperparameter Tuning:

Original: The hyperparameters are tuned using Bayes optimization functionality of Weights&Biases 4 by evaluating in-distribution validation accuracy.
Revised: The hyperparameters are tuned using Bayes optimization functionality of Weights&Biases 4 by evaluating in-distribution validation accuracy.

Original: Bayesian optimization sequentially explored the potential hyperparameter candidate points, and we evaluated all the trained models in the search process for comparison.
Revised: Bayesian optimization sequentially explored the potential hyperparameter candidate points, and we evaluated all the trained models in the search process for comparison.

Original: As a confirmation of the soundness of our hyperparameter search, Appendix
Revised: As a confirmation of the soundness of our hyperparameter search, Appendix

Original: J shows the histogram that the explored hyperparameters are drawn from reasonably wide range.
Revised: J.1 shows the histogram that the explored hyperparameters are drawn from reasonably wide range.

Original: The data shown in the box plot (Figure 3, 4, and 5) are from the evaluation of several trained models.
Revised: The data shown in the box plot (Figure 3, 4, and 5) are from the evaluation of several trained models.

Original: The number of epochs (the steps budget) used is in line with previous studies [GL21; Koh+21; Cho+19] to ensure the soundness of our experimental design.
Revised: The number of epochs (the steps budget) used is in line with previous studies [GL21; Koh+21; Cho+19] to ensure the soundness of our experimental design.

Original: Since we use the fixed epoch, it might seem to be unfair than when tuning the epoch as well for the optimizer that converges faster.
Revised: Since we use the fixed epoch, it might seem to be unfair than when tuning the epoch as well for the optimizer that converges faster.

Original: However, we studied the effect of early stopping, which corresponds to the tuned epoch in Appendix I.4 and the result confirms that employing a fixed epoch does not impair the fairness of our comparison experiments.
Revised: However, we studied the effect of early stopping, which corresponds to the tuned epoch in Appendix I.4 and the result confirms that employing a fixed epoch does not impair the fairness of our comparison experiments.

Original: We discuss the details in Appendix I.4.
Revised: We discuss the details in Appendix I.4.

Original: Boxplot:
Revised: Boxplot:

Original: We believe that sharing the whole distribution of tuning outcomes is important because: it gives an idea of how sensitive methods are to tuning and how much tuning effort is required.
Revised: We believe that sharing the whole distribution of tuning outcomes is important because: it gives an idea of how sensitive methods are to tuning and how much tuning effort is required.

Original: We also share the raw data as scatter plots (Figure 22, 23, and 24) in Appendix H.3.
Revised: We also share the raw data as scatter plots (Figure 22, 23, and 24) in Appendix H.3.

Original: Model Selection Method and Evaluation Metrics:
Revised: Model Selection Method and Evaluation Metrics:

Original: In the training phase of DomainBed datasets, we do not access the data in the test domain but split data from the training domains into a training dataset and validation dataset.
Revised: In the training phase of DomainBed datasets, we do not access the data in the test domain but split data from the training domains into a training dataset and validation dataset.

Original: We choose the model with the highest average performance (accuracy) on the validation data in the training domain.
Revised: We choose the model with the highest average performance (accuracy) on the validation data in the training domain.

Original: As a metric for evaluation, we evaluated the generalization performance in the test domain as the OOD accuracy.
Revised: As a metric for evaluation, we evaluated the generalization performance in the test domain as the OOD accuracy.

Original: The Backgrounds Challenge uses Imagenet-1k as the training data set and selects models based on their accuracy on the validation data in the training domain.
Revised: The Backgrounds Challenge uses Imagenet-1k as the training data set and selects models based on their accuracy on the validation data in the training domain.

Original: After that, we measure the in-distribution performance with IN9L [Xia+21], which aggregates the test data into nine classes.
Revised: After that, we measure the in-distribution performance with IN9L [Xia+21], which aggregates the test data into nine classes.

Original: As for the OOD performance, we measure the classification performance on the data where the background image of
Revised: As for the OOD performance, we measure the classification performance on the data where the background image of

Original: IN9L is replaced with the background image of other images.
Revised: IN9L is replaced with the background image of other images.

Original: In CivilComments-WILDS, we divide the data into training, validation, and test datasets and maximize worst-group accuracy in the validation data (and by association, maximize the average accuracy over all domains).
Revised: In CivilComments-WILDS, we divide the data into training, validation, and test datasets and maximize worst-group accuracy in the validation data (and by association, maximize the average accuracy over all domains).

Original: Then, we perform model selection and evaluate the OOD accuracy on the test data.
Revised: Then, we perform model selection and evaluate the OOD accuracy on the test data.

Original: For Amazon-WILDS datasets, we adopt the same hyperparameter selection strategy as that for
Revised: For Amazon-WILDS datasets, we adopt the same hyperparameter selection strategy as that for

Original: CivilComments-WILDS.
Revised: CivilComments-WILDS.

Original: As a metric, we do not evaluate the worst-group performance but rather the 10th-percentile accuracy for the performance of each domain by following the standard federated learning literature [Cal+18].
Revised: As a metric, we do not evaluate the worst-group performance but rather the 10th-percentile accuracy for the performance of each domain by following the standard federated learning literature [Cal+18].

Original: In line with previous studies ([GL21], [Xia+21], and [Koh+21]), different benchmarks use different evaluation metrics, each of which is outlined in the following sections.
Revised: In line with previous studies ([GL21], [Xia+21], and [Koh+21]), different benchmarks use different evaluation metrics, each of which is outlined in the following sections.

Original: E.1 DomainBed
Revised: E.1 DomainBed

Original: We follow the setting that is employed in the original paper [GL21].
Revised: We follow the setting that is employed in the original paper [GL21].

Original: We train models with training domains and evaluate their performance on the test domain, which is the domain not used for training.
Revised: We train models with training domains and evaluate their performance on the test domain, which is the domain not used for training.

Original: We use ResNet-50 for PACS, VLCS, Office-Home, DomainNet and TerraIncognita and MNIST
Revised: We use ResNet-50 for PACS, VLCS, Office-Home, DomainNet and TerraIncognita and MNIST

Original: ConvNet [GL21] for RotatedMNIST and Colored MNIST.
Revised: ConvNet [GL21] for RotatedMNIST and Colored MNIST.

Original: In our experiments, one domain is used as the test domain (out-of-distribution) and the other domains as the training domain (in-distribution).
Revised: In our experiments, one domain is used as the test domain (out-of-distribution) and the other domains as the training domain (in-distribution).

Original: More precisely, the test domain is "Art" for PACS, "Caltech101" for VLSC, "Art" for Office-Home, "Clipart" for DomainNet, "L100" for TerraIncognita, " 30 °" for RotatedMNIST, and "-90%" for ColoredMNIST.
Revised: More precisely, the test domain is "Art" for PACS, "Caltech101" for VLSC, "Art" for Office-Home, "Clipart" for DomainNet, "L100" for TerraIncognita, " 30 °" for RotatedMNIST, and "-90%" for ColoredMNIST.

Original: We explain the experimental configurations in Section F.1.
Revised: We explain the experimental configurations in Section F.1.

Original: E.2 Backgrounds Challenge Dataset
Revised: E.2 Backgrounds Challenge Dataset

Original: We follow [Xia+21] for using Backgrounds Challenge dataset for evaluation.
Revised: We follow [Xia+21] for using Backgrounds Challenge dataset for evaluation.

Original: Thus, we use the following data-generating procedure proposed by [Xia+21].
Revised: Thus, we use the following data-generating procedure proposed by [Xia+21].

Original: We train ResNet-50 on ImageNet-1k with two popular optimizers, Momentum SGD, and Adam.
Revised: We train ResNet-50 on ImageNet-1k with two popular optimizers, Momentum SGD, and Adam.

Original: The test datasets to evaluate the trained model are derivations of ImageNet dataset.
Revised: The test datasets to evaluate the trained model are derivations of ImageNet dataset.

Original: First, we construct a subset of the ImageNet which has nine coarse-grained classes, e.g. insect ( ORIGINAL ).
Revised: First, we construct a subset of the ImageNet which has nine coarse-grained classes, e.g. insect ( ORIGINAL ).

Original: Especially, we refer to ORIGINAL with all images from ImageNet as IN9L .
Revised: Especially, we refer to ORIGINAL with all images from ImageNet as IN9L .

Original: Then, we create a dataset by changing the background of the images of IN9L. In particular, we change the background of each image into a random background cropped from another image in ORIGINAL.
Revised: Then, we create a dataset by changing the background of the images of IN9L. In particular, we change the background of each image into a random background cropped from another image in ORIGINAL.

Original: We call this dataset Mixed-Random , following [Xia+21].
Revised: We call this dataset Mixed-Random , following [Xia+21].

Original: By comparing the accuracy of Mixed-Same with that of ORIGINAL, we can measure the dependence of the model on the spurious correlation of background information.
Revised: By comparing the accuracy of Mixed-Same with that of ORIGINAL, we can measure the dependence of the model on the spurious correlation of background information.

Original: Thus, we investigate the relations between these two accuracies.
Revised: Thus, we investigate the relations between these two accuracies.

Original: The search range for hyperparameters is shown in Section F.2.
Revised: The search range for hyperparameters is shown in Section F.2.

Original: E.3 WILDS
Revised: E.3 WILDS

Original: We also follow the setting that is employed in the original paper [Koh+21].
Revised: We also follow the setting that is employed in the original paper [Koh+21].

Original: First, we divide the dataset into train, validation, and test and train a model using the train data.
Revised: First, we divide the dataset into train, validation, and test and train a model using the train data.

Original: For model selection, we use the performance evaluation of validation data.
Revised: For model selection, we use the performance evaluation of validation data.

Original: In the test dataset, considering the subpopulation shift, we measure the performance of the OOD in the worst group for CivilComments-WILDS and in the domain of 10-percentile for Amazon-WILDS.
Revised: In the test dataset, considering the subpopulation shift, we measure the performance of the OOD in the worst group for CivilComments-WILDS and in the domain of 10-percentile for Amazon-WILDS.

Original: In both Amazon-WILDS and CivilComments-WILDS,
Revised: In both Amazon-WILDS and CivilComments-WILDS,

Original: DistilBERT [San+19] is used as the deep neural network model architecture.
Revised: DistilBERT [San+19] is used as the deep neural network model architecture.

Original: We explain the further details of the experimental configurations in Section F.3.
Revised: We explain the further details of the experimental configurations in Section F.3.

Original: F Hyperparameters and Detailed Configurations
Revised: F Hyperparameters and Detailed Configurations

Original: We report the hyperparameter’s search space.
Revised: We report the hyperparameter’s search space.

Original: For vanilla SGD, we search learning rate η , learning rate decay rate ρ and the timing to decay learning rate δ , and regularization coefficient of weight decay λ .
Revised: For vanilla SGD, we search learning rate η , learning rate decay rate ρ and the timing to decay learning rate δ , and regularization coefficient of weight decay λ .

Original: When δ = 0 .
Revised: When δ = 0 .

Original: 7 , it means that the learning rate decays when training passes 70 % of the total iterations.
Revised: 7 , it means that the learning rate decays when training passes 70 % of the total iterations.

Original: We do not search ρ and δ for DomainBed because we do not employ a learning rate schedule.
Revised: We do not search ρ and δ for DomainBed because we do not employ a learning rate schedule.

Original: For non-adaptive momentum methods, a parameter to control momentum γ is added to the hyperparameters.
Revised: For non-adaptive momentum methods, a parameter to control momentum γ is added to the hyperparameters.

Original: For RMSprop, we further add parameters α and ϵ , which control the second-order momentum and numerical stability, respectively.
Revised: For RMSprop, we further add parameters α and ϵ , which control the second-order momentum and numerical stability, respectively.

Original: Although ϵ is originally introduced for numerical stability, this parameter is found to play a crucial role in generalization performance [Cho+19].
Revised: Although ϵ is originally introduced for numerical stability, this parameter is found to play a crucial role in generalization performance [Cho+19].

Original: Thus, we follow Choi et al.
Revised: Thus, we follow Choi et al.

Original: and vary this parameter as well.
Revised: and vary this parameter as well.

Original: For Adam, we add ϵ , β 1 , β 2 to vanilla SGD’s configuration.
Revised: For Adam, we add ϵ , β 1 , β 2 to vanilla SGD’s configuration.

Original: The parameter ϵ is the same as that for
Revised: The parameter ϵ is the same as that for

Original: RMSprop and β 1 and β 2 control first and second-order momentum terms, respectively.
Revised: RMSprop and β 1 and β 2 control first and second-order momentum terms, respectively.

Original: F.1 DomainBed
Revised: F.1 DomainBed

Original: We conduct Bayesian optimization for hyperparameter search of DomainBed.
Revised: We conduct Bayesian optimization for hyperparameter search of DomainBed.

Original: First, we sampled hyperparameters from uniform distribution whose minimum and maximum values are shown in the table 5 and 

Revised: First, we sampled hyperparameters from uniform distribution whose minimum and maximum values are shown in the table 5 and

Original: Then, we conducted Bayesian optimization on these sampled candidate hyperparameters and selected some of the hyperparameters among them.
Revised: Then, we conducted Bayesian optimization on these sampled candidate hyperparameters and selected some of the hyperparameters among them.

Original: Note that the values for batch size B in the tables do not indicate minimum and maximum for Bayesian optimization but those for grid search.
Revised: Note that the values for batch size B in the tables do not indicate minimum and maximum for Bayesian optimization but those for grid search.

Original: Unlike other datasets, we implement IRMv1 in addition to ERM.
Revised: Unlike other datasets, we implement IRMv1 in addition to ERM.

Original: IRMv1 is a heuristic optimization problem to solve IRM, introduced by Arjovsky et al [Arj+19].
Revised: IRMv1 is a heuristic optimization problem to solve IRM, introduced by Arjovsky et al [Arj+19].

Original: Thus, we search the hyperparameters of IRMv1 as well.
Revised: Thus, we search the hyperparameters of IRMv1 as well.

Original: IRMv1 is the following constrained optimization problem [Arj+19]:
Revised: IRMv1 is the following constrained optimization problem [Arj+19]:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where Φ is representation function and w is the weight on top of the function of a model f = w ◦ Φ .
Revised: where Φ is representation function and w is the weight on top of the function of a model f = w ◦ Φ .

Original: X is the input domain and Y is the output domain.
Revised: X is the input domain and Y is the output domain.

Original: The character e indicates an environment in training environment set E tr
Revised: The character e indicates an environment in training environment set E tr

Original: and R e is the risk of the environment.
Revised: and R e is the risk of the environment.

Original: Because this is the optimization with regularization term, we search coefficient λ IRM .
Revised: Because this is the optimization with regularization term, we search coefficient λ IRM .

Original: In addition, we implement annealing of this coefficient and so we try various penalty annealing iterations N IRM too.
Revised: In addition, we implement annealing of this coefficient and so we try various penalty annealing iterations N IRM too.

Original: The basic workload is summarized in table 4.
Revised: The basic workload is summarized in table 4.

Original: We made two changes to the experimental setup in the original paper, as well as to the optimizer.
Revised: We made two changes to the experimental setup in the original paper, as well as to the optimizer.

Original: We added regularization to the training of ColoredMNIST and RotatedMNIST to stabilize the learning.
Revised: We added regularization to the training of ColoredMNIST and RotatedMNIST to stabilize the learning.

Original: In addition, we increased the steps budget for RotatedMNIST because we observed that the training loss of RotatedMNIST was not sufficiently reduced.
Revised: In addition, we increased the steps budget for RotatedMNIST because we observed that the training loss of RotatedMNIST was not sufficiently reduced.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F.2 Backgrounds Challenge Dataset
Revised: F.2 Backgrounds Challenge Dataset

Original: We conduct Bayesian optimization for the Backgrounds Challenge dataset as well.
Revised: We conduct Bayesian optimization for the Backgrounds Challenge dataset as well.

Original: We use 4096 as the batch size.
Revised: We use 4096 as the batch size.

Original: For all configurations other than batch size, as we follow Choi et al [Cho+19].
Revised: For all configurations other than batch size, as we follow Choi et al [Cho+19].

Original: We conduct the hyperparameter search and restart training from scratch.
Revised: We conduct the hyperparameter search and restart training from scratch.

Original: Of the trained models, we evaluate trained model performance in the OOD dataset
Revised: Of the trained models, we evaluate trained model performance in the OOD dataset

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F.3 WILDS
Revised: F.3 WILDS

Original: We conduct Bayesian optimization for the hyperparameter search of WILDS.
Revised: We conduct Bayesian optimization for the hyperparameter search of WILDS.

Original: The hyperparameters are sampled by uniform distribution whose minimum and maximum values are shown in the table 8 and 9.
Revised: The hyperparameters are sampled by uniform distribution whose minimum and maximum values are shown in the table 8 and 9.

Original: Note that the values for batch size B is fixed in these experiments due to computational efficiency.
Revised: Note that the values for batch size B is fixed in these experiments due to computational efficiency.

Original: We implement IRMv1 as well as DomainBed experiments.
Revised: We implement IRMv1 as well as DomainBed experiments.

Original: For training the Amazon-WILDS and CivilComments-WILDS datasets that we used as NLP datasets, we followed the deep neural network model and training configuration proposed in the original paper.
Revised: For training the Amazon-WILDS and CivilComments-WILDS datasets that we used as NLP datasets, we followed the deep neural network model and training configuration proposed in the original paper.

Original: DistilBERT, a distillation of BERT Base, is used as the deep neural network model.[Table] [Table]
Revised: DistilBERT, a distillation of BERT Base, is used as the deep neural network model.

Original: [1e-6, 1e-2] [0, 0.99999] - - [1e-1, 1e+5] [1e+0, 1e+4]

Original: Nesterov [16] [1e-5, 1e-2] [1e-6, 1e-2] [0, 0.99999] - - [1e-1, 1e+5] [1e+0, 1e+4]

Original: RMSprop [16] [1e-5, 1e-2] [1e-6, 1e-2] [0, 0.99999] [

Original: [Table]
Revised: [Table]

Original: [1e

Original: [Table]

Original: [Table]

Original: 1e

Original: [Table]

Original: ] [1e

Original: [Table]

Original: 1e

Original: [Table]

Original: ] [1e

Original: [Table]

Original: [Table]

Original: Adam [16] [1e

Original: [Table]
Revised: [Table]

Original: 1e

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: 1e

Original: [Table]

Original: [Table]

Original: [Table]

Original: G Main Results G.1 Experimental Results on DomainBed
Revised: G Main Results G.1 Experimental Results on DomainBed

Original: Figure 3 shows a box plot of the difference between the average in-distribution accuracy and average
Revised: Figure 3 shows a box plot of the difference between the average in-distribution accuracy and average

Original: OOD accuracy for ERM.
Revised: OOD accuracy for ERM.

Original: In the following discussion, we call this difference a gap for convenience.
Revised: In the following discussion, we call this difference a gap for convenience.

Original: Due to the limitation of the paper length, only the plots of PACS and Office-Home are shown here.
Revised: Due to the limitation of the paper length, only the plots of PACS and Office-Home are shown here.

Original: All results, including IRM results, are shown in Appendixes H.1.1 and H.1.2.
Revised: All results, including IRM results, are shown in Appendixes H.1.1 and H.1.2.

Original: We found two distinct optimizer effect patterns, depending on the dataset: i) PACS, Office-Home,
Revised: We found two distinct optimizer effect patterns, depending on the dataset: i) PACS, Office-Home,

Original: VLCS, Terra Incognita, Rotated MNIST, and DomainNet and ii) Colored MNIST.
Revised: VLCS, Terra Incognita, Rotated MNIST, and DomainNet and ii) Colored MNIST.

Original: Because these patterns appear in both the results of ERM and IRM, we focus on ERM for the explanation.
Revised: Because these patterns appear in both the results of ERM and IRM, we focus on ERM for the explanation.

Original: For PACS, Office Home, VLCS, TerraIncognita, RotatedMNIST, and DomainNet, the OOD accuracy is greater for non-adaptive optimizers.
Revised: For PACS, Office Home, VLCS, TerraIncognita, RotatedMNIST, and DomainNet, the OOD accuracy is greater for non-adaptive optimizers.

Original: The gap between the mean OOD accuracy and the mean in-distribution accuracy was smaller for the non-adaptive optimizer except for TerraIncognita.
Revised: The gap between the mean OOD accuracy and the mean in-distribution accuracy was smaller for the non-adaptive optimizer except for TerraIncognita.

Original: This means that the models trained with the non-adaptive optimizer are more robust on average.
Revised: This means that the models trained with the non-adaptive optimizer are more robust on average.

Original: In
Revised: In

Original: TerraIncognita, the non-adaptive optimizer significantly outperforms the adaptive optimizer on the average in-distribution performance and OOD performance.
Revised: TerraIncognita, the non-adaptive optimizer significantly outperforms the adaptive optimizer on the average in-distribution performance and OOD performance.

Original: We note, however, that except in this problem, the adaptive optimizer achieves a smaller gap between the two.
Revised: We note, however, that except in this problem, the adaptive optimizer achieves a smaller gap between the two.

Original: As shown in Figure 1, when comparing models with the same in-distribution performance, the non-adaptive optimizer showed higher OOD accuracy than the adaptive optimizer.
Revised: As shown in Figure 1, when comparing models with the same in-distribution performance, the non-adaptive optimizer showed higher OOD accuracy than the adaptive optimizer.

Original: The results in Figures 1 and 3 confirm that the non-adaptive method achieves higher OOD accuracy than the adaptive method, both on average and for models with the same in-distribution performance.
Revised: The results in Figures 1 and 3 confirm that the non-adaptive method achieves higher OOD accuracy than the adaptive method, both on average and for models with the same in-distribution performance.

Original: Colored MNIST shows the opposite pattern of these results, where the adaptive optimizer is better than the non-adaptive optimizers.
Revised: Colored MNIST shows the opposite pattern of these results, where the adaptive optimizer is better than the non-adaptive optimizers.

Original: As explained in Section 2.1, Colored MNIST is distinct from the other datasets.
Revised: As explained in Section 2.1, Colored MNIST is distinct from the other datasets.

Original: To understand why adaptive optimizers are better at OOD generalization on this dataset, we plot the time development of the average training accuracy, validation accuracy, and average test accuracy across training, as shown in Figure 35.
Revised: To understand why adaptive optimizers are better at OOD generalization on this dataset, we plot the time development of the average training accuracy, validation accuracy, and average test accuracy across training, as shown in Figure 35.

Original: We explain this result in Appendix I.3.
Revised: We explain this result in Appendix I.3.

Original: We note our considerations regarding the exceptional behavior of ColoredMNIST.
Revised: We note our considerations regarding the exceptional behavior of ColoredMNIST.

Original: ColoredMNIST is a binary classification task dataset with random labels, where spurious features are positively correlated with invariant features in in-distribution and negatively correlated with OOD.
Revised: ColoredMNIST is a binary classification task dataset with random labels, where spurious features are positively correlated with invariant features in in-distribution and negatively correlated with OOD.

Original: Non-adaptive optimizers learn the spurious features, achieve the oracle performance for in-distribution and perform worse than a random guess for OOD due to inverted correlations.
Revised: Non-adaptive optimizers learn the spurious features, achieve the oracle performance for in-distribution and perform worse than a random guess for OOD due to inverted correlations.

Original: Adaptive optimizers, in contrast, seem to overfit the training data, achieving 100% accuracy in the training set (more than oracle) in this dataset, and failing to learn the spurious features.
Revised: Adaptive optimizers, in contrast, seem to overfit the training data, achieving 100% accuracy in the training set (more than oracle) in this dataset, and failing to learn the spurious features.

Original: Due the aforementioned (synthetic) inverted correlations in the dataset, this overfitting behaviour in-distribution happens to favor adaptive optimizers.
Revised: Due the aforementioned (synthetic) inverted correlations in the dataset, this overfitting behaviour in-distribution happens to favor adaptive optimizers.

Original: This behavior happens exceptionally on ColoredMNIST due to these synthetic flips in correlation.
Revised: This behavior happens exceptionally on ColoredMNIST due to these synthetic flips in correlation.

Original: This exploitation unexpectedly enables Adam to avert being trapped in the training domain and produces better OOD generalization.
Revised: This exploitation unexpectedly enables Adam to avert being trapped in the training domain and produces better OOD generalization.

Original: We would like to study this possibility further in future research.
Revised: We would like to study this possibility further in future research.

Original: [Figure]
Revised: [Figure]

Original: PACS
Revised: PACS

Original: [Figure]
Revised: [Figure]

Original: ERM
Revised: ERM

Original: [Figure]
Revised: [Figure]

Original: G.2 Experimental Results on Backgrounds Challenge
Revised: G.2 Experimental Results on Backgrounds Challenge

Original: Backgrounds Challenge requires training of ImageNet-1k on ResNet50 from scratch, and it takes 256 GPU hours to obtain a single trained model, so we only compared Momentum SGD with Adam.
Revised: Backgrounds Challenge requires training of ImageNet-1k on ResNet50 from scratch, and it takes 256 GPU hours to obtain a single trained model, so we only compared Momentum SGD with Adam.

Original: Because Momentum SGD achieved competitive performance among non-adaptive optimization methods in the DomainBed and WILDS experiments, we adopted Momentum SGD to represent non-adaptive optimization methods.
Revised: Because Momentum SGD achieved competitive performance among non-adaptive optimization methods in the DomainBed and WILDS experiments, we adopted Momentum SGD to represent non-adaptive optimization methods.

Original: In the same way, Adam outperformed RMSProp in all the experiments, so we adopted Adam as representative of the adaptive optimizers.
Revised: In the same way, Adam outperformed RMSProp in all the experiments, so we adopted Adam as representative of the adaptive optimizers.

Original: Figure 4 compares the accuracy for ORIGINAL (in-distribution) and Mixed-Rand (out-ofdistribution).
Revised: Figure 4 compares the accuracy for ORIGINAL (in-distribution) and Mixed-Rand (out-ofdistribution).

Original: The best in-distribution performance is the same for each optimizer, but concerning the best OOD performance, the non-adaptive optimizer outperformed the adaptive optimizer.
Revised: The best in-distribution performance is the same for each optimizer, but concerning the best OOD performance, the non-adaptive optimizer outperformed the adaptive optimizer.

Original: As can be seen from Figure 1 (second row, middle column), particularly in the region of high in-distribution performance on the right, the OOD performance of Momentum SGD exceeds that of Adam.
Revised: As can be seen from Figure 1 (second row, middle column), particularly in the region of high in-distribution performance on the right, the OOD performance of Momentum SGD exceeds that of Adam.

Original: [Figure]
Revised: [Figure]

Original: G.3 Experimental Results on WILDS
Revised: G.3 Experimental Results on WILDS

Original: A comparison of in-distribution and OOD averages for WILDS is shown in Figure 5.
Revised: A comparison of in-distribution and OOD averages for WILDS is shown in Figure 5.

Original: It can be clearly seen that the adaptive optimizer is fit too well to the in-distribution in the WILDS problem setting.
Revised: It can be clearly seen that the adaptive optimizer is fit too well to the in-distribution in the WILDS problem setting.

Original: For Amazon-WILDS and CivilComment-WILDS, as in DomainBed and Backgrounds Challenge, the non-adaptive optimizer outperforms the adaptive optimizer in terms of OOD generalization.
Revised: For Amazon-WILDS and CivilComment-WILDS, as in DomainBed and Backgrounds Challenge, the non-adaptive optimizer outperforms the adaptive optimizer in terms of OOD generalization.

Original: The gap between in-distribution accuracy and OOD accuracy is also tiny for non-adaptive optimizers.
Revised: The gap between in-distribution accuracy and OOD accuracy is also tiny for non-adaptive optimizers.

Original: In particular, the CivilComments-WILDS experimental result is remarkable, as both non-adaptive and adaptive optimizers show similar high in-distribution performance
Revised: In particular, the CivilComments-WILDS experimental result is remarkable, as both non-adaptive and adaptive optimizers show similar high in-distribution performance

Original: [Figure]
Revised: [Figure]

Original: but in the OOD environment, adaptive methods significantly fail to make inferences
Revised: but in the OOD environment, adaptive methods significantly fail to make inferences

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: G.4 Experimental Results: Correlation Behaviors
Revised: G.4 Experimental Results: Correlation Behaviors

Original: The increasing return is an example, as shown in the leftmost part of Figure 2.
Revised: The increasing return is an example, as shown in the leftmost part of Figure 2.

Original: The increasing returns in large regions of the in-distribution generalization significantly affect the OOD generalization, suggesting that the last tuning is significant for the OOD generalization, as seen in all domain generalization problems except for DomainNet.
Revised: The increasing returns in large regions of the in-distribution generalization significantly affect the OOD generalization, suggesting that the last tuning is significant for the OOD generalization, as seen in all domain generalization problems except for DomainNet.

Original: The linear return is as shown in the middle of Figure 2.
Revised: The linear return is as shown in the middle of Figure 2.

Original: The OOD accuracy increases linearly with the in-distribution accuracy.
Revised: The OOD accuracy increases linearly with the in-distribution accuracy.

Original: This is generally the same result for in-distribution validation and test.
Revised: This is generally the same result for in-distribution validation and test.

Original: The diminishing return is an example like that shown in the rightmost part of Figure 2, where the increase in a large region of the in-distribution is saturating the OOD generalization with a small percentage of its effect.
Revised: The diminishing return is an example like that shown in the rightmost part of Figure 2, where the increase in a large region of the in-distribution is saturating the OOD generalization with a small percentage of its effect.

Original: This behavior suggests that the effort of tuning at the end is often not worth it to improve OOD generalization.
Revised: This behavior suggests that the effort of tuning at the end is often not worth it to improve OOD generalization.

Original: In addition to Amazon-WILDS, we have seen a similar trend in problem settings with subpopulation shifts, such as CivilComments-WILDS.
Revised: In addition to Amazon-WILDS, we have seen a similar trend in problem settings with subpopulation shifts, such as CivilComments-WILDS.

Original: H Full Results of Experiments H.1 Full Results of Boxplot
Revised: H Full Results of Experiments H.1 Full Results of Boxplot

Original: Since we could not include all the results in the main paper
Revised: Since we could not include all the results in the main paper

Original: [Figure]
Revised: [Figure]

Original: we report all the OOD acc, In-distribution acc and their difference (we denote as gap) results including ERM and IRM results in a box plot.
Revised: we report all the OOD acc, In-distribution acc and their difference (we denote as gap) results including ERM and IRM results in a box plot.

Original: H.1.1 Full Results of Filtered Boxplot
Revised: H.1.1 Full Results of Filtered Boxplot

Original: What we want to find out is the OOD generalization performance for models that perform better than random guess in the In-distribution test set, i
Revised: What we want to find out is the OOD generalization performance for models that perform better than random guess in the In-distribution test set, i

Original: [Figure]
Revised: [Figure]

Original: e., models that have been trained.
Revised: e., models that have been trained.

Original: As Filtered
Revised: As Filtered

Original: Results
Revised: Results

Original: [Figure]
Revised: [Figure]

Original: we define random guess = 1 / num of classes in each problem setting
Revised: we define random guess = 1 / num of classes in each problem setting

Original: [Figure]
Revised: [Figure]

Original: and show the results of eliminating those models whose in-distribution accuracy does not exceed this threshold
Revised: and show the results of eliminating those models whose in-distribution accuracy does not exceed this threshold

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: cc
Revised: cc

Original: PACS: IRM
Revised: PACS: IRM

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: PACS: IRM
Revised: PACS: IRM

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: VLCS
Revised: VLCS

Original: [Table]
Revised: [Table]

Original: IRM
Revised: IRM

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table] [Figure] [Table]
Revised: [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Figure]
Revised: [Figure]

Original: RotatedMNIST
Revised: RotatedMNIST

Original: [Figure]
Revised: [Figure]

Original: RotatedMNIST
Revised: RotatedMNIST

Original: [Figure]
Revised: [Figure]

Original: IRM
Revised: IRM

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t
Revised: t

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: H.1.2 Full Results of Non-Filtered Boxplot
Revised: H.1.2 Full Results of Non-Filtered Boxplot

Original: In the filtered results, we excluded the evaluation of models that failed to learn, but if we want to discuss the possibility of learning failure
Revised: In the filtered results, we excluded the evaluation of models that failed to learn, but if we want to discuss the possibility of learning failure

Original: [Figure]
Revised: [Figure]

Original: it is important to look at the unfiltered data.
Revised: it is important to look at the unfiltered data.

Original: Therefore
Revised: Therefore

Original: [Figure]
Revised: [Figure]

Original: we also created a box plot for models that performed below a random guess, without excluding them
Revised: we also created a box plot for models that performed below a random guess, without excluding them

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: OfficeHome: ERM
Revised: OfficeHome: ERM

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: - d i s t r i b
Revised: - d i s t r i b

Original: [Figure]
Revised: [Figure]

Original: i o n a cc
Revised: i o n a cc

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: cc
Revised: cc

Original: PACS: IRM
Revised: PACS: IRM

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original:  [Figure] [Table]
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: PACS: IRM
Revised: PACS: IRM

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: VLCS
Revised: VLCS

Original: [Table]
Revised: [Table]

Original: IRM
Revised: IRM

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] [Table] 
Revised: [Figure] [Table] [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: i s
Revised: i s

Original: [Figure]
Revised: [Figure]

Original: r i
Revised: r i

Original: [Figure]
Revised: [Figure]

Original: u t i o n a cc
Revised: u t i o n a cc

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: H.3 Full Results of Scatter Plots
Revised: H.3 Full Results of Scatter Plots

Original: The results of the comparison of OOD accuracy and in-distribution accuracy shown in Section 3. are shown below for all benchmarks.
Revised: The results of the comparison of OOD accuracy and in-distribution accuracy shown in Section 3. are shown below for all benchmarks.

Original: The box plots shown in Section H.1 and the Reliability Diagram Like Plot shown in Section H.2 are based on the data shown in the Scatter Plot shown below.
Revised: The box plots shown in Section H.1 and the Reliability Diagram Like Plot shown in Section H.2 are based on the data shown in the Scatter Plot shown below.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: o
Revised: o

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: i s t
Revised: i s t

Original: r i b u t i o n a cc
Revised: r i b u t i o n a cc

Original: [Table]
Revised: [Table]

Original: t r i b u t i o n a cc
Revised: t r i b u t i o n a cc

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: u
Revised: u

Original: [Figure]
Revised: [Figure]

Original: o f - d i s t r i b u t i o n a cc
Revised: o f - d i s t r i b u t i o n a cc

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: t r i b u t i o n a cc
Revised: t r i b u t i o n a cc

Original: [Table]
Revised: [Table]

Original: t r i b u t i o n a cc
Revised: t r i b u t i o n a cc

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: u
Revised: u

Original: [Figure]
Revised: [Figure]

Original: - o f - d i s t r i b u t i o n a cc
Revised: - o f - d i s t r i b u t i o n a cc

Original: [Figure]
Revised: [Figure]

Original: IRM
Revised: IRM

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: cc
Revised: cc

Original: WILDS_amazon:IRM
Revised: WILDS_amazon:IRM

Original: [Figure]
Revised: [Figure]

Original: I Additional Results of Experiments
Revised: I Additional Results of Experiments

Original: In addition to the experimental results presented in the main paper, we provide a more in-depth analysis, which is shown in this chapter.
Revised: In addition to the experimental results presented in the main paper, we provide a more in-depth analysis, which is shown in this chapter.

Original: First of all, We show that the pattern of linear correlation of accuracy claimed by [Mil+21] does not necessarily occur even when the correlation patterns we observed, such as diminishing returns, are probit transformed.
Revised: First of all, We show that the pattern of linear correlation of accuracy claimed by [Mil+21] does not necessarily occur even when the correlation patterns we observed, such as diminishing returns, are probit transformed.

Original: The results of the probit transform of the scatter plots shown in Appendix H.3 are shown, following the method of [Mil+21].
Revised: The results of the probit transform of the scatter plots shown in Appendix H.3 are shown, following the method of [Mil+21].

Original: This may be due to the fact that our experimental setup uses a larger number of data sets and takes IRM and other factors into account.
Revised: This may be due to the fact that our experimental setup uses a larger number of data sets and takes IRM and other factors into account.

Original: In the second section, we present results comparing the performance improvement of OOD with a larger trial budget in the search for hyperparameters by Bayesian optimization.
Revised: In the second section, we present results comparing the performance improvement of OOD with a larger trial budget in the search for hyperparameters by Bayesian optimization.

Original: Thirdly, We investigate why Adam shows better OOD performance than SGD in the negatively correlated case seen in Figure 7 for ColoredMNIST by considering the learning curve.
Revised: Thirdly, We investigate why Adam shows better OOD performance than SGD in the negatively correlated case seen in Figure 7 for ColoredMNIST by considering the learning curve.

Original: Finally, we examine the effectiveness of the early stopping for each optimizer to study if adaptive optimizers overfit because of their speed of convergence.
Revised: Finally, we examine the effectiveness of the early stopping for each optimizer to study if adaptive optimizers overfit because of their speed of convergence.

Original: I.1 Probit Transformed Scatter Plot
Revised: I.1 Probit Transformed Scatter Plot

Original: As shown in Section 3.3, the work of [Mil+21] compares the accuracy of OOD with the accuracy of in-distribution by showing a plot on a probit scale.
Revised: As shown in Section 3.3, the work of [Mil+21] compares the accuracy of OOD with the accuracy of in-distribution by showing a plot on a probit scale.

Original: Their comparison shows that the correlation of accuracy is linear.
Revised: Their comparison shows that the correlation of accuracy is linear.

Original: In this section, we convert the scatter plots identified in Section H.3 to probit scale and confirm that they do not necessarily show a linear correlation (Figure 27, 28, 29, 30, 30, and 31).
Revised: In this section, we convert the scatter plots identified in Section H.3 to probit scale and confirm that they do not necessarily show a linear correlation (Figure 27, 28, 29, 30, 30, and 31).

Original: I.1.1 ERM
Revised: I.1.1 ERM

Original: [Figure]
Revised: [Figure]

Original: PACS:ERM scaling)
Revised: PACS:ERM scaling)

Original: [Figure]
Revised: [Figure]

Original: VLCS
Revised: VLCS

Original: [Figure]
Revised: [Figure]

Original: (probit scaling)
Revised: (probit scaling)

Original: [Figure]
Revised: [Figure]

Original: OfficeHome:ERM (probit scaling
Revised: OfficeHome:ERM (probit scaling

Original: [Figure]
Revised: [Figure]

Original: :ERM (probit scaling
Revised: :ERM (probit scaling

Original: [Figure]
Revised: [Figure]

Original: :ERM (probit scaling)
Revised: :ERM (probit scaling)

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: I.1.2 IRM
Revised: I.1.2 IRM

Original: [Figure]
Revised: [Figure]

Original: PACS:IRM (probit scaling)
Revised: PACS:IRM (probit scaling)

Original: [Figure]
Revised: [Figure]

Original: VLCS
Revised: VLCS

Original: [Figure]
Revised: [Figure]

Original: probit
Revised: probit

Original: [Figure]
Revised: [Figure]

Original: OfficeHome:IRM (probit scaling)
Revised: OfficeHome:IRM (probit scaling)

Original: [Figure]
Revised: [Figure]

Original: TerraIncognita:IRM (probit scaling)
Revised: TerraIncognita:IRM (probit scaling)

Original: [Figure]
Revised: [Figure]

Original: RotatedMNIST:IRM (probit scaling) in
Revised: RotatedMNIST:IRM (probit scaling) in

Original: [Figure]
Revised: [Figure]

Original: distribution
Revised: distribution

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: I.3 Learning Curve of ColoredMNIST
Revised: I.3 Learning Curve of ColoredMNIST

Original: In Section G.1, we conjecture that the better performance of Adam in Colored MNIST classification may come from overfitting to training data.
Revised: In Section G.1, we conjecture that the better performance of Adam in Colored MNIST classification may come from overfitting to training data.

Original: To confirm this hypothesis, we plot the averaged training accuracy, the averaged validation accuracy, and the averaged test accuracy throughout the training.
Revised: To confirm this hypothesis, we plot the averaged training accuracy, the averaged validation accuracy, and the averaged test accuracy throughout the training.

Original: We pick the top-14 results in terms of test accuracy and use them for the plot.
Revised: We pick the top-14 results in terms of test accuracy and use them for the plot.

Original: We show the result in
Revised: We show the result in

Original: Figure
Revised: Figure

Original: [Figure]
Revised: [Figure]

Original: As is evident from Figure 35 (a) and Figure 35 (b)
Revised: As is evident from Figure 35 (a) and Figure 35 (b)

Original: [Figure]
Revised: [Figure]

Original: we observe that training accuracy increases while the validation accuracy keeps unchanged.
Revised: we observe that training accuracy increases while the validation accuracy keeps unchanged.

Original: This indicates that overfitting occurs in the training.
Revised: This indicates that overfitting occurs in the training.

Original: However, Figure 35 (c) indicates that test accuracy gradually improves as the model is overfitting.
Revised: However, Figure 35 (c) indicates that test accuracy gradually improves as the model is overfitting.

Original: On the other hand, SGD shows no such overfitting and OOD generalization improvement, as shown in Figure 36.
Revised: On the other hand, SGD shows no such overfitting and OOD generalization improvement, as shown in Figure 36.

Original: Thus, we can empirically support our conjecture that Adam produces the better OOD generalization performance by overfitting training data.
Revised: Thus, we can empirically support our conjecture that Adam produces the better OOD generalization performance by overfitting training data.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: a v
Revised: a v

Original: [Figure]
Revised: [Figure]

Original: r a g e t e s t a cc u r a c y
Revised: r a g e t e s t a cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: a g
Revised: a g

Original: [Figure]
Revised: [Figure]

Original: t r a i n a cc u r a c y
Revised: t r a i n a cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: of checkpoints whose train accuracy is
Revised: of checkpoints whose train accuracy is

Original: in the top-
Revised: in the top-

Original: I.4 Early Stopping
Revised: I.4 Early Stopping

Original: In Section G.1, figures 1 shows that adaptive optimizers tend to overfit to training domain.
Revised: In Section G.1, figures 1 shows that adaptive optimizers tend to overfit to training domain.

Original: A possible reason is that the training speed of adaptive methods is faster than non-adaptive ones.
Revised: A possible reason is that the training speed of adaptive methods is faster than non-adaptive ones.

Original: That is, in the same steps budget, adaptive optimizers converge faster in effect.
Revised: That is, in the same steps budget, adaptive optimizers converge faster in effect.

Original: To validate if this is the case, we investigate whether early stopping improves the OOD generalization of adaptive optimizers.
Revised: To validate if this is the case, we investigate whether early stopping improves the OOD generalization of adaptive optimizers.

Original: In particular, we compute the difference between averaged accuracy at early stopping and at last epoch for test accuracy and validation accuracy, respectively:
Revised: In particular, we compute the difference between averaged accuracy at early stopping and at last epoch for test accuracy and validation accuracy, respectively:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: If the average accuracy at early stopping is larger, the difference is positive and vice versa.
Revised: If the average accuracy at early stopping is larger, the difference is positive and vice versa.

Original: Figures 37 to 42 are the results of this comparison for each dataset and algorithm.
Revised: Figures 37 to 42 are the results of this comparison for each dataset and algorithm.

Original: The y-axis is the difference of test accuracy and the x-axis is the difference of validation accuracy.
Revised: The y-axis is the difference of test accuracy and the x-axis is the difference of validation accuracy.

Original: The color indicates the epoch of early stopping.
Revised: The color indicates the epoch of early stopping.

Original: The darker color indicates that early stopping is conducted in relatively earlier epochs.
Revised: The darker color indicates that early stopping is conducted in relatively earlier epochs.

Original: For PACS, both differences are positive and lighter colors are concentrated at points of small difference of the validation accuracy, as indicated in Figures 37 and 38.
Revised: For PACS, both differences are positive and lighter colors are concentrated at points of small difference of the validation accuracy, as indicated in Figures 37 and 38.

Original: Therefore, we can conclude that when the validation accuracy deteriorates by further training, the test accuracy also gets worse.
Revised: Therefore, we can conclude that when the validation accuracy deteriorates by further training, the test accuracy also gets worse.

Original: However, the effect of further training is less evident for Adam.
Revised: However, the effect of further training is less evident for Adam.

Original: In other words, early stopping does not influence Adam so much but keeps test accuracy from decreasing for SGD.
Revised: In other words, early stopping does not influence Adam so much but keeps test accuracy from decreasing for SGD.

Original: The result of VLCS shows a similar pattern as PACS (Figures 39 and 40).
Revised: The result of VLCS shows a similar pattern as PACS (Figures 39 and 40).

Original: If anything, Further training after early stopping makes adaptive optimizes be likely to result in better test accuracy than SGD, though they have a large variance.
Revised: If anything, Further training after early stopping makes adaptive optimizes be likely to result in better test accuracy than SGD, though they have a large variance.

Original: With respect to SGD, additional training degrades the test accuracy as much as it degrades validation accuracy.
Revised: With respect to SGD, additional training degrades the test accuracy as much as it degrades validation accuracy.

Original: Office-Home shows similar results as PACS, as presented in Figures 41 and 42.
Revised: Office-Home shows similar results as PACS, as presented in Figures 41 and 42.

Original: In summary, we find that early stopping does not influence adaptive optimizers.
Revised: In summary, we find that early stopping does not influence adaptive optimizers.

Original: Therefore, we can conclude that adaptive optimizer overfits not because it trains faster than non-adaptive methods.
Revised: Therefore, we can conclude that adaptive optimizer overfits not because it trains faster than non-adaptive methods.

Original: The fact that early stopping does not affect the adaptive optimizer means that adjusting the number of epochs instead of a fixed number of epochs will not change the result.
Revised: The fact that early stopping does not affect the adaptive optimizer means that adjusting the number of epochs instead of a fixed number of epochs will not change the result.

Original: We have followed previous studies and experimented with a fixed number of epochs in the present study, and our results suggest that this does not have a serious impact on the comparison of adaptive and non-adaptive optimizers.
Revised: We have followed previous studies and experimented with a fixed number of epochs in the present study, and our results suggest that this does not have a serious impact on the comparison of adaptive and non-adaptive optimizers.

Original: Thus, we can see that using a fixed epoch number does not undermine the validity of our optimizer comparison experiment in this sense.
Revised: Thus, we can see that using a fixed epoch number does not undermine the validity of our optimizer comparison experiment in this sense.

Original: [Figure]
Revised: [Figure]

Original: difference
Revised: difference

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: n c
Revised: n c

Original: [Figure]
Revised: [Figure]

Original: o
Revised: o

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: a cc u r a c y
Revised: a cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: difference of validation accuracy
Revised: difference of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: validation
Revised: validation

Original: accuracy
Revised: accuracy

Original: [Figure]
Revised: [Figure]

Original: cc
Revised: cc

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: difference of validation accuracy
Revised: difference of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: difference of validation accuracy
Revised: difference of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: d i ff e r e n c e o f t e s t a cc u r a c
Revised: d i ff e r e n c e o f t e s t a cc u r a c

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: difference of
Revised: difference of

Original: [Figure]
Revised: [Figure]

Original: ff e
Revised: ff e

Original: [Figure]
Revised: [Figure]

Original: e n c e o f t e s t a cc u r a c y
Revised: e n c e o f t e s t a cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: difference of validation accuracy
Revised: difference of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: difference
Revised: difference

Original: of validation accuracy
Revised: of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: ff
Revised: ff

Original: [Figure]
Revised: [Figure]

Original: e n c e o
Revised: e n c e o

Original: [Figure]
Revised: [Figure]

Original: e s
Revised: e s

Original: [Figure]
Revised: [Figure]

Original: a cc u
Revised: a cc u

Original: [Figure]
Revised: [Figure]

Original: a c
Revised: a c

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: difference
Revised: difference

Original: of validation accuracy
Revised: of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: difference
Revised: difference

Original: [Figure]
Revised: [Figure]

Original: accuracy
Revised: accuracy

Original: [Figure]
Revised: [Figure]

Original: d i ff e r e n c e o f t e s t a cc u r a c
Revised: d i ff e r e n c e o f t e s t a cc u r a c

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: n c e o f t e s t a cc u r a c y
Revised: n c e o f t e s t a cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: validation accuracy
Revised: validation accuracy

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: s
Revised: s

Original: [Figure]
Revised: [Figure]

Original: a cc u
Revised: a cc u

Original: [Figure]
Revised: [Figure]

Original: a c
Revised: a c

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: difference of validation accuracy
Revised: difference of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: d i ff e r e n c e o f t e s t a cc u r a c
Revised: d i ff e r e n c e o f t e s t a cc u r a c

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ff e
Revised: ff e

Original: [Figure]
Revised: [Figure]

Original: e n c e o f t e s t a cc u r a c y
Revised: e n c e o f t e s t a cc u r a c y

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: e n c e o
Revised: e n c e o

Original: [Figure]
Revised: [Figure]

Original: e s
Revised: e s

Original: [Figure]
Revised: [Figure]

Original: a cc u
Revised: a cc u

Original: [Figure]
Revised: [Figure]

Original: a c
Revised: a c

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: d i ff e r e n c e o f t e s t a cc u r a
Revised: d i ff e r e n c e o f t e s t a cc u r a

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: n c e o f t e s t a cc u r a c y
Revised: n c e o f t e s t a cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: difference of validation accuracy
Revised: difference of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: of validation accuracy
Revised: of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: s
Revised: s

Original: [Figure]
Revised: [Figure]

Original: a cc u
Revised: a cc u

Original: [Figure]
Revised: [Figure]

Original: a c
Revised: a c

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: of validation accuracy
Revised: of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: difference of validation accuracy
Revised: difference of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: d i ff e r e n c e o f t e s t a cc u r a c y adam sorted by percentage
Revised: d i ff e r e n c e o f t e s t a cc u r a c y adam sorted by percentage

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: n c e o f t e s t a cc u r a c y
Revised: n c e o f t e s t a cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: a cc u
Revised: a cc u

Original: [Figure]
Revised: [Figure]

Original: a c
Revised: a c

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: difference of validation accuracy
Revised: difference of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: d i ff e r e n c e o f t e s t a cc u r a
Revised: d i ff e r e n c e o f t e s t a cc u r a

Original: [Figure]
Revised: [Figure]

Original: However, we find different results from Colored MNIST.
Revised: However, we find different results from Colored MNIST.

Original: As shown in Figures 43 and 44, we can observe that the difference of validation accuracy is positive and that of test accuracy is negative.
Revised: As shown in Figures 43 and 44, we can observe that the difference of validation accuracy is positive and that of test accuracy is negative.

Original: That is further training after early stopping decreases validation accuracy but increases test accuracy on
Revised: That is further training after early stopping decreases validation accuracy but increases test accuracy on

Original: Colored MNIST, while there is no difference for SGD.
Revised: Colored MNIST, while there is no difference for SGD.

Original: This is consistent with the result of Appendix
Revised: This is consistent with the result of Appendix

Original: I.3.
Revised: I.3.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ff e
Revised: ff e

Original: [Figure]
Revised: [Figure]

Original: e n c e o
Revised: e n c e o

Original: [Figure]
Revised: [Figure]

Original: t e
Revised: t e

Original: [Figure]
Revised: [Figure]

Original: t a cc u r a c y
Revised: t a cc u r a c y

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ff e
Revised: ff e

Original: [Figure]
Revised: [Figure]

Original: e n c e o f t e s t a cc
Revised: e n c e o f t e s t a cc

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: d i ff e r e n c e o f t e s t a cc u r a c
Revised: d i ff e r e n c e o f t e s t a cc u r a c

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: d i ff e r e n c e o f t e s t a cc u r a c y
Revised: d i ff e r e n c e o f t e s t a cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: difference of validation accuracy
Revised: difference of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: accuracy
Revised: accuracy

Original: [Figure]
Revised: [Figure]

Original: d i ff e r e n c e o f t e s t a cc u
Revised: d i ff e r e n c e o f t e s t a cc u

Original: [Table]
Revised: [Table]

Original: a c
Revised: a c

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: of validation accuracy
Revised: of validation accuracy

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: d i ff e r e n c e o f t e s t a cc u r a
Revised: d i ff e r e n c e o f t e s t a cc u r a

Original: [Figure]
Revised: [Figure]

Original: J  Histgram of Hyperparameters
Revised: J Soundness Check of Our Experiments J.1 Histgram of Hyperparameters

Original: We have shown the histgram of the hyperparameters to be used for training just for reference.
Revised: We have shown the histgram of the hyperparameters to be used for training just for reference.

Original: In particular, we display a result for learning rate of Momentum SGD and Adam for the each dataset.
Revised: In particular, we display a result for learning rate of Momentum SGD and Adam for the each dataset.

Original: We observe that we could sample hyperparameters from a reasonably wide range.
Revised: We observe that we could sample hyperparameters from a reasonably wide range

Original: [Figure]
Revised: [Figure]

Original: (learning rate) in training of DomainBed, WILDS dataset in ERM. Momentum SGD and Adam results are included.
Revised: In order to confirm the soundness of our experiments

Original: Although uniform distribution is used as theprior distribution for hyperparameter optimization, the histogram results do not match the uniform distribution because Bayesian optimization is used. K The Number ofTrials
Revised: [Table] we compared our results with existing benchmarks to see how well they actually performed, in addition to the hyperparameter search ranges in the previous section. In particular, we compared our experimental results with those of

Original: For reference
Revised: DomainBed [GL21], an existing benchmark that uses Adam.

Original: [Table]
Revised: [Table]

Revised: J.3 The Number of Trials for Hyperparameter Search

Original:  we show the table of the number of trials for each dataset/optimizer combination on
Revised: For reference, we show the table of the number of trials for each dataset/optimizer combination on

Original: ERM in Table 10.
Revised: ERM in Table 11.

Original: [Table]
Revised: [Table]