Improving Baselines in the Wild
Abstract
We share our experience with the recently released WILDS benchmark, a collection of ten datasets dedicated to developing models and training strategies which are robust to domain shifts.
Several experiments yield a couple of critical observations which we believe are of general interest for any future work on WILDS.
Our study focuses on two datasets: iWildCam and FMoW.
We show that (1) Conducting separate cross-validation for each evaluation metric is crucial for both datasets, (2) A weak correlation between validation and test performance might make model development difﬁcult for iWildCam, (3) Minor changes in the training of hyperparameters improve the baseline by a relatively large margin (mainly on FMoW), (4) There is a strong correlation between certain domains and certain target labels (mainly on iWildCam).
To the best of our knowledge, no prior work on these datasets has reported these observations despite their obvious importance. Our code is public.
Introduction
It goes without saying that common benchmark datasets and solid baselines are essential ingredients for developing new models and correctly measuring progress in machine learning.
Key datasets such as ImageNet [1] in image recognition, Switchboard [2] in automatic speech recognition, or Penn
Treebank [3, 4] in language modelling, to name only a few, have played a crucial role in demonstrating the impressive performance of neural networks (NNs) especially in the 2010s.
They have contributed to gradually marking the progress obtained by new techniques and models.
Equally importantly, strong baselines are crucial to properly measure progress.
In some cases, by revisiting some old or standard baseline conﬁgurations, we end up with surprisingly good results, including results of LSTMs for language modelling [5], ResNets for vision [6] or Transformers for systematic generalization [7].
Nowadays, few researchers are surprised by NNs performing very well on some in-domain data distribution, and there has been increasing interest in developing models that are robust to domain shifts [8, 9, 10, 11].
Here we focus on the recently proposed benchmark for evaluating domain robust systems, WILDS [11], and we share our empirical experience with two datasets of WILDS, iWildCam and FMoW, as well as their baseline models.
A handful of experiments exhibit important, previously unreported facets of the data and the baselines.
Our main ﬁndings can be summarised as follows: (1) Conducting separate cross-validation for each evaluation metric is crucial for both datasets, (2) A weak correlation between validation and test performance might make model development difﬁcult (iWildCam), (3) Minor changes in the training hyper-parameters improve the baseline by a relatively large margin (mainly on FMoW), (4) There is a strong correlation between certain domains and certain target labels (mainly on iWildCam).
Any practitioner should be aware of these aspects when developing new ideas based on these two datasets, as well as on other datasets of WILDS.
[Table]
Datasets and Experimental Settings
Here we brieﬂy summarize essential properties of the datasets used in our experiments.
For further information, we refer to the original paper by Koh et al.
iWildCam2020-WILDS.
I W ILD C AM 2020-WILDS (iWildCam for short) is a variant of the iWildCam 2020 Competition Dataset [12].
The task is classiﬁcation of photos of 182 different animal species from various camera traps.
The camera traps deﬁne various domains of this dataset.
The training set contains about 130 K photos taken by 243 camera traps.
The out-of-distribution validation and test sets consist of photos from 32 and 48 different camera traps which are not part of the training domains.
Following Koh et al.
[11], we report the classiﬁcation accuracy, as well as the macro F score which Koh et al.
[11] set as the most relevant evaluation metric.
FMoW-WILDS.
FMoW-WILDS (FMoW for short) is a variant of the functional map of the world dataset [13].
This is also an image classiﬁcation task using satellite images containing one ofbuilding or land classes.
There are two attributes for grouping these images: years and geographical regions (Africa, the Americas, Oceania, Asia, or Europe).
The training data consist of images taken before 2013, the validation from 2013 to 2015, and the test set from 2016.
Koh et al.
[11] set the worst region accuracy (WR) as the main evaluation metric, and also report the total accuracy (TA).
Experimental settings.
Unless otherwise stated, we use all default settings provided by the publicly available ofﬁcial codebase for WILDS [11].
Core ﬁndings
In this section, we highlight our ﬁndings on I W ILD C AM 2020- WILDS (iWildCam for short) and
FM O W- WILDS (FMoW for short).
We note that all our results are produced using the ofﬁcial baseline code provided by Koh et al.
[11] with small modiﬁcations speciﬁed below.
Metric-wise cross-validation is crucial.
In both iWildCam and FMoW, the performance of models is measured on two or more evaluation metrics (one of them is considered to be the “main” metric). E.g., for iWildCam, the classiﬁcation accuracy (AC) and macro F1 (F1) are reported on in-domain (ID) and out-of-domain (OOD) subsets of validation and test sets.
Assuming that we mainly care about the OOD performance, this results in two metrics to monitor during the model development:
OOD validation AC and F1.
Technically, this implies a necessity for conducting cross-validation and checkpoint tracking separately for each metric.
We note that such a metric-wise cross-validation is not part of the ofﬁcial setting [11].
Furthermore, we observe that in the ofﬁcial setting, the crossvalidation is carried out only at the end of each epoch.
We increase the cross-validation frequency (to every 1000 steps) such that we do not miss good checkpoints between epochs.
Table 1 shows the performance of different checkpoints selected using two different CV criteria within the same training runs on iWildCam.
The ﬁrst thing to notice here is that simply introducing checkpoint tracking
Following Koh et al.
[11], we report mean and std dev over 3 training runs.
[Table]
between epochs can improve the baseline performance by a non-negligible margin.
In Table [Table] if we compare “Our run” which uses the default setting by Koh et al.
[11] and “+ freq ckpt” which does cross validation every 1000 training steps: the OOD validation accuracy improves from 64.7% to 66.6% and the F1 from 36.2% to 38.3%.
The impact of metric-wise CV is also relatively large:
In our best conﬁguration (“+ freq ckpt” in Table 1), the AC-CV checkpoint (i [Table] e. the best checkpoint found by cross validation based on OOD validation accuracy) achieves an OOD validation accuracy of 66.6% vs. 64.1% for the F1-CV one.
Similarly, the
F1-CV model achieves an OOD validation F1 score of 38.3% compared to 36.6% for the AC-CV one.
We stress that these performance gaps are obtained without any technical changes on the algorithmic level, revealing that very careful tuning is needed to compare models on this dataset.
Strictly speaking, we should also do separate cross-validations on the ID validation metrics to report and compare the ID performance.
We omit this here as our results on the OOD metrics already clearly exhibit the issue.
While our goal is not to develop separate models for different metrics, we observe how crucial it is to report the cross-validation setting for fair comparisons between different approaches.
Correlation between validation and test performance is weak.
Another important observation we drawn from Table 1 is that despite a relatively large performance gap between the F1-CV and
AC-CV models in terms of OOD validation accuracy (64.1 vs. 66.6%), their OOD test accuracy is similar (69.0 vs. 68.6%).
This trend is less visible for FMoW (Table 2, batch size 64), but still:
our
WR-CV model outperforms the ofﬁcial baseline on the WR validation accuracy (50.2 vs. 48.9%) while it underperforms on the corresponding WR test accuracy (31.4 vs. 32.3%).
This is actually not surprising as the validation and test sets are also OOD to each other, hence improvements on the validation set do not necessarily transfer to the test set.
However, this is problematic since we only have access to the OOD validation performance while developing models.
The common practice of selecting the checkpoint based on a validation set (also recommended by the original work [11]; in the paragraph on avoiding overﬁtting to the test sets) might not work here: we thus have no reliable reference metric for developing new models.
A similar problem regarding the lack of useful validation sets has been recently pointed out [7] in the context of systematic generalization [14].
This problem seems even harder for general domain shifts: unlike in the case of systematic generalization, there is no a priori way of controlling the degree of "domain shifts" systematically.
This calls for discussing the construction of useful validation sets when studying OOD generalization.
There is sub-optimality in the baseline.
Another crucial observation regarding FMoW is the suboptimality of the baseline setting.
While keeping all conﬁgurations equal to the original conﬁguration [11], we modify the training batch size in the FMoW baseline.
As is shown in Table 2, simply by reducing the batch size from 64 to 20, the best/worst region accuracy (WR; the main evaluation metric for FMoW) improves from 50.2% to 52.4% on the OOD validation set, and from 31.4% to 34.4% on the OOD test set.
By further modifying the learning rate from 1e-4 to 3e-4 and increasing the checkpoint frequency to be every 200 steps instead of 1000, we ﬁnally obtain a worst region accuracy of 34.8% (rows “+ higher lr”).
We note that these ﬁnal OOD test accuracies we obtain are competitive compared to the number reported 2 by Shi et al.
[9], 34.6%, achieved using a technique speciﬁcally designed for the domain shift problem (a method called Fish; last row in Table 2).
Such “trivial” improvements over the baseline call for further tuning of the baseline settings before developing and evaluating new models on this dataset.
Correlation between domains and target labels is rather strong.
Our last observation to be shared here is that certain domains have a bias towards certain target labels.
Each image in iWildCam is labeled with a target class label (one of 182 animal species) and a domain label (one of 323 camera traps).
For each domain, we count the number of unique class labels covered within the domain i.e., if this number is 1 for a certain domain, all images belonging to that domain have the same target class label.
Figure 1 shows the cumulative histogram of the corresponding statistics.
As can be seen, the number of unique class labels (x-axis) vary only from 1 to 24 (while the maximum is 182) and the 50% of the domains contain less than 5 distinct classes (indicated by the yellow line).
This number improves to 10 distinct classes only when we take 75% of the domains (red line).
Overall, the diversity of target class labels is very limited within each domain.
This bias can potentially be problematic when developing training strategies which make use of some grouping of datapoints by the domain label.
A number of techniques has been proposed to exploit domain information [15, 16, 17] for domain robust learning.
Interestingly, none of these techniques have resulted in consistent improvements over the basic empirical risk minimisation baseline [11].
We point out these previously unreported important statistical properties although further analysis is required to draw conclusions regarding the causal relationship between these two observations.
Conclusion
To properly evaluate new machine learning methods and measure progress, it is crucial to start from strong and well-established baselines.
We reported a couple of important observations from our experiments with the baseline settings of WILDS.
The latter can be improved by simply tweaking a few conﬁgurations (frequent metric-wise cross-validation) and one hyper-parameter for FMoW (batch size).
Our observations seem to indicate that a systematic study of baseline conﬁgurations is necessary before starting the development of new models based on WILDS.
We hope that practitioners will ﬁnd our observations useful, and take them into account for future work on WILDS.
Acknowledgments and Disclosure of Funding
This research was partially funded by ERC Advanced grant no: 742870, project AlgoRNN, and by
Swiss National Science Foundation grant no: 200021_192356, project NEUSYM.
This work was partially supported by computational resources at the CSCS Swiss National Supercomputing Centre, project d115.
We thank NVIDIA Corporation for donating several DGX machines, and IBM for donating a Minsky machine.