S CALABLE T RANSFER L EARNING WITH E XPERT M ODELS
Research
Google Research
Sylvain
A BSTRACT
Transfer of pre-trained representations can improve sample efﬁciency and reduce computational requirements for new tasks.
However, representations used for transfer are usually generic, and are not tailored to a particular distribution of downstream tasks.
We explore the use of expert representations for transfer with a simple, yet effective, strategy.
We train a diverse set of experts by exploiting existing label structures and use cheap-to-compute performance proxies to select the relevant expert for each target task.
This strategy scales the process of transferring to new tasks, since it does not revisit the pre-training data during transfer.
Accordingly, it requires little extra compute per target task, and results in a speed-up of 2– orders of magnitude compared to competing approaches.
Further, we provide an adapter-based architecture able to compress many experts into a single model.
We evaluate our approach on two different data sources and demonstrate that it outperforms baselines on over 20 diverse vision tasks in both cases
I NTRODUCTION
Deep learning has been successful on many computer vision tasks.
Unfortunately, this success often requires a large amount of per-task data and compute.
To scale deep learning to new vision tasks, practitioners often turn to transfer learning.
Transfer learning involves re-using models trained on a large source task, and tuning them on the target task.
This can improve both convergence rates (Ben-David et al., 2007; 2010; Blitzer et al., 2008; Du et al., 2017; Kuzborskij & Orabona, 2013;
Mansour et al., 2009) and empirical performance (Dai et al., 2007; Donahue et al., 2014; Oquab et al., 2014; Tan et al., 2018).
Transfer learning reduces per-task data or compute requirements, given a large one-off pre-training cost.
In practice, this one-off down payment may not be made by the practitioner, since pre-trained networks are made available through platforms like PyTorch and TensorFlow Hub 1 .
For instance, ImageNet pre-training is popular since it is freely available and works well for many tasks (Donahue et al., 2014; Oquab et al., 2014; Sharif Razavian et al., 2014).
In contrast to generic homogeneous models (e.g. most pre-trained ImageNet networks), Mixture of
Experts (MoE) include multiple heterogeneous sub-models (“experts”) that specialize to sub-problems of the full task.
MoEs have been studied for decades (Eigen et al., 2013; Jacobs & Jordan, 1993), and have also been successful in deep learning (Shazeer et al., 2017).
Yet, the application of experts for deep transfer learning has been less explored.
We study visual transfer with experts, and present a simple, scalable, yet effective strategy.
Transfer of specialist models has been studied before.
However, they either require expensive retraining on the source dataset for every target task (Ngiam et al., 2018; Yan et al., 2020), or operate at a small scale where all experts can be applied simultaneously (Dvornik et al., 2020).
Further, most of them are tested only on a limited suite of natural single-object classiﬁcation tasks.
We lift these
Step 3.
Given a new downstream task D T
=
( X T , Y T ) , we compute the image representations
M e ( X T ) from each expert e .
We use kNN to compute the accuracy on the supervised problem
D T,e = ( M e ( X T ) , Y T ) , and select the expert e ∗ with highest accuracy.
Step 4.
We add a new head to e ∗ and ﬁne-tune its whole network with the downstream data, leading to the ﬁnal model.
constraints, and present a practical approach that scales to hundreds of large experts, while requiring relatively little compute per target task .
Our strategy consists of four stages (ﬁg. 1).
(1) Unconditional pre-training .
A single baseline model is trained on the entire upstream data.
(2) Experts training .
Multiple experts are pre-trained by exploiting the label hierarchy present in many large-scale image datasets, such as ImageNet and JFT.
In addition to entire expert networks, we explore residual adapters that allow all of the expertise to be packed into a single model that can be loaded into memory.
These two stages may be expensive, but are done only once.
(3) Expert selection .
Applying all experts to each task does not scale well; some sort of sparsiﬁcation is required.
We focus on inexpensive model selection that can be applied to hundreds or thousands of experts.
(4) Downstream ﬁne-tuning .
We take the output of the model selection phase and tune it on the target task.
Importantly, this phase does not require revisiting the source dataset, which may be unavailable or expensive to train on.
We show that this approach yields remarkably strong performance on many diverse tasks.
We evaluate not only on classic vision tasks, but also on the diverse VTAB benchmark of 19 tasks (Zhai et al., 2019).
Our contributions can be summarized as follows.
• We propose a transfer learning algorithm with a large number of experts based on per-task routing via nearest neighbors selection.
Once we have amortized the pre-training cost, this algorithm requires little compute per target task , achieving an speed-up of 500 × – 1000 × compared to competing strategies.
Also, it can be easily replicated with any large upstream multilabel dataset.
• We achieve a mean accuracy improvement of 3.6% over the state-of-the-art performance onVTAB datasets using ResNet50 networks.
Our algorithm offers improvements on every group of tasks: natural, specialized, and structured.
Figure 2 summarizes these results.
• We explore using sub-networks as experts via residual adapters, allowing all experts to be packed into a single model.
Surprisingly these perform almost as well as their full-network counterparts.
R ELATED W ORK
Transfer Learning.
Tasks with little training data can beneﬁt from other larger datasets, often from a similar domain.
Transfer learning concerns the link between the source and target dataset (Pan & Yang, 2009;
Weiss et al., 2016; Tan et al., 2018; Wang, 2018).
One family of methods creates a single training dataset, where source instances are re-weighted according to their relevance (Dai et al., 2007; Pardoe & Stone, 2010; Wan et al., 2011; Xu et al., 2017).
A popular method consists of ﬁne-tuning a model that was pre-trained on the source data (Donahue et al., 2014; Oquab et al., 2014; Sharif Razavian et al., 2014).
Some transfer learning algorithms condition the initial source model on the target dataset itself (Ngiam et al., 2018; Xie et al., 2019; Yalniz et al., 2019), while
A v e r a g e a cc u r a c y 69.
All VTAB     77.
Natural     81.6 83.7 83.6 84.
Specialized     57.2 58.6 58.8 59.
Structured
[Figure]
others (like ours) are agnostic about the downstream task when the initial model is trained on the source data (Kolesnikov et al., 2019).
We offer an in-depth comparison with (Ngiam et al., 2018) in section 6.6.
In the context of few-shot learning, where out-of-the-box ﬁne-tuning may not work, generic representations are sometimes frozen, and simple feature selection (Dvornik et al., 2020) or model training (Chen et al., 2019) techniques are applied on top.
Instead of relying on ﬁxed universal representations, (Rebufﬁ et al., 2017; 2018) use small additional modules, or adapters, that incorporate knowledge from several visual domains.
Our work also explores this idea.
Multi-task Learning.
MTL tries to leverage the common aspects of several learning tasks (Caruana, 1997).
A prominent approach uses explicit parameter sharing; for instance, by means of common low-level layers leading to different heads.
Among others, this has been successfully applied to vision (Zhang et al., 2014), language (Liu et al., 2015), and reinforcement learning (Fedus et al., 2019) tasks.
In addition, a variety of ways to combine task-speciﬁc representations have arisen, such as cross-stitch networks (Misra et al., 2016), or lateral connections (Rusu et al., 2016).
A different family of methods impose joint constraints on the –possibly different– models corresponding to each task.
We can combine the learning problems via regularization and shared sparsity patterns (Argyriou et al., 2007; Lounici et al., 2009), or by imposing some prior knowledge regarding the task structure (Evgeniou et al., 2005; Jacob et al., 2009; Kim et al., 2012).
T HE T RANSFER L EARNING F RAMEWORK
In this section, we describe our transfer learning setup of interest.
The high-level goal is to train strong models for arbitrary downstream tasks, possibly under severe data and compute limitations.
To do so efﬁciently, one can ofﬂoad computation to a previous upstream phase which is executed a priori , without knowing the downstream tasks in advance.
Accordingly, the upstream model should not depend on any speciﬁc target data.
We are mostly interested in the low data regime where downstream tasks contain few datapoints.
These restrictions have a practical motivation: we would like to build and deploy universal representations that are easily transferred to a wide range of downstream settings.
Any transfer algorithm must implement the following three stages.
Upstream Training.
Given the upstream data D U , the algorithm ﬁrst outputs a source model M .
The goal is to provide useful initial representations for various new tasks.
This stage could actually produce a family of models { M e } rather than a single one.
These models might not be disjoint, and could share parameters.
The upstream learning problems are auxiliary; accordingly, D U could include a diverse set of classiﬁcation, regression, or even synthetic learning instances.
Model Selection.
When a new downstream task is given, a selection algorithm is applied to choose the upstream model(s) to transfer, possibly depending on the downstream data.
This phase should be computationally cheap; thus, the upstream data is no longer available.
Sometimes, there is no choice to make (say, with a single ImageNet representation).
Alternatively, in models with a complex structure, one may choose which parts, routes, or modules to keep in a data-dependent fashion.
Downstream Training.
The ﬁnal stage ﬁne-tunes the selected model using the downstream data, either fully or partially.
For neural nets, a new head is added as the output classes are task-speciﬁc.
Our overall algorithm is depicted in ﬁg. 1.
We give details about each step in the following sections.
[Figure]
U PSTREAM T RAINING 4. E XPERT A RCHITECTURES
Our experts should provide feature extractions that are a good starting point to learn future tasks related to the expert’s upstream training data.
We explore two different model architectures to train such experts.
As an obvious choice, we ﬁrst look at ResNets (He et al., 2016b).
These are powerful models; however, storing and deploying many of them can be challenging.
As an alternative, we also develop more compact adapter modules that can all be assembled in a single architecture.
Also, their individual size can be easily customized to meet memory and computational constraints, which makes them an ideal candidate for combining multiple experts in a single model, when needed.
We informally refer to these as full and adapter modules (or experts), respectively.
Full ResNet Modules.
As a base architecture for full experts we use ResNets.
In particular, all of our experiments focus on the ResNet50-v2 architecture (R50) (He et al., 2016a), which sequentially stacks a root block and 4 blocks with (3, 4, 6, 3) residual units.
The initial step in every experiment consists of training a baseline model B on the whole upstream data (see stage 1 in ﬁg. 1).
This baseline is subsequently ﬁne-tuned by both full and adapter experts, but in different ways.
A full expert trained on a slice of data is simply the baseline B ﬁne-tuned on that data.
The head will later be discarded for transfer.
This approach requires as many R50s as there are experts.
Adapter Modules.
Residual adapters were proposed to adapt a neural network to a particular downstream task without needing to ﬁne-tune the entire network (Rebufﬁ et al., 2017).
Instead, we use them to adapt the baseline architecture to slices of the upstream data .
Originally, they were 1 ×convolutions placed after each 3 × 3 convolution, with a residual connection.
Instead we place them before each of the R50’s blocks.
Finally, our adapters have a bottleneck and are non-linear, as in (Houlsby et al., 2019).
We insert several in parallel into the backbone B .
When creating an expert, only the adapters are tuned and the backbone weights are frozen.
Figure 3a depicts the ResNet architecture with multiple expert adapters ( a ( i ) 1 , . .
, a ( i ) n ).
Let F i be the function implemented by the i -th block of the backbone network.
We adapt its input by computing the output as x i :=
F i ( x i − 1 + a ( i ) e ( x i − 1 )) , where e = R ( x ) is the identity of the selected expert, given by some routing function R , and x is the original input.
During upstream training, the function
R may also use the labels in addition to the image, as we discuss in section 4.3.
Figure 3b shows the adapter’s bottleneck architecture.
An adapter sequentially applies components Aand A 2 .
Each component performs a group normalization (N) (Wu & He, 2018), a ReLU activation (A) (Glorot et al., 2011), and a convolution (C) (LeCun et al., 1989), in that order.
Due to the skip connection, the output dimension of A 2 ◦ A 1 must match that of its input, c .
However, we can change the output channels k of A 1 , in order to limit the amount of parameters.
Thus, we set k = c
2 so that the number of parameters equals that of a linear adapter.
Each adapter only increases the parameter count of the R50 backbone by 6%.
We brieﬂy explored placing these adapters in other locations, or using other variations (Rebufﬁ et al., 2018), but we did not observe any signiﬁcant improvement.
It is important to emphasize that the names Full and Adapters refer to the parameters that are specialized to a given subset of the upstream data before any downstream ﬁne-tuning .
In both cases, we ﬁne-tune the entire network, after the model selection phase, to the downstream task.
4. U PSTREAM D ATA AND E XPERT D EFINITION
We train our upstream models on large vision datasets with thousands of classes.
Moreover, the datasets include an expressive hierarchy, linking classes and ancestor concepts via “is-a” relationships.
Our experts’ domains are nodes in this hierarchy, which are selected automatically based on the number of images.
Due to the multi-label nature of the datasets, several experts could simultaneously apply to an image.
For example, for an image of a lion, all of organism, animal, carnivore, felidae , and lion could be relevant expert domains.
In particular, we use two different upstream image datasets, and independently train a set of experts on each.
We further describe them in section 6.1.
4. E XPERT T RAINING
Recall we denote by B the baseline R50 model trained on the whole upstream dataset D U .
As shown in ﬁg. 1, the second step of upstream training consists of training each expert individually on different subsets of the upstream dataset.
Let D e :=
( X e , Y e ) ⊆
D U be the data corresponding to expert e .
The subsets corresponding to different experts may overlap (e.g. for the animal and dog experts).
As mentioned before, the full experts completely ﬁne-tune B on D e .
For the adapter experts the weights corresponding to the adapter e (modules in red in ﬁg. 3) are trained on D e , but the shared blocks and head parameters are frozen .
Note that, due to the sharing scheme, we can train all experts independently in parallel.
We train all experts for the same number of steps, regardless of the size of
D e .
Instead of learning a routing function, we exploit the structure of the upstream labels and use a hard-coded routing.
We found this makes learning easier, and leads to powerful specialized models.
E XPERT S ELECTION
Given a new downstream dataset D T =
( X T , Y T ) , we must choose an expert to use.
We consider three approaches: domain prediction , label matching , and performance proxy .
Domain Prediction.
This strategy selects the expert solely based on the images X T .
It effectively selects the expert whose domain best matches the target images.
We implement this by training an auxiliary network (the “Expert Prediction Network” or EPN) to classify the expert from the image (i.e. learn the hard-coded routing mentioned previously).
The EPN is trained upstream using the pre-training data and expert assignments.
During transfer, an expert is selected using the highest geometric mean EPN predictive probability across the dataset.
Details are in the Appendix A.
Label Matching.
Matching the task with an expert can be done in the label space as opposed to the input space, by computing the afﬁnity of each expert to a new downstream task in the label space of the upstream dataset.
We ﬁrst use a generic network trained on all upstream labels to predict upstream labels on the downstream images.
We compute the KL-divergence between the distribution of labels on the downstream task images, and the prior distribution of labels for each expert.
This per-expert prior is computed as the empirical distribution of labels on the images used to train that expert.
We select the expert with the smallest KL-divergence.
Details are in the Appendix B.
Performance Proxy.
The aforementioned two strategies are simple, but do not use the training labels
Y T available for downstream tasks, which may contain key information.
It would be too expensive to ﬁne-tune every expert to every new task and select the best with hindsight, so we propose a proxy for the ﬁnal performance.
For this, we use a k -nearest neighbors classiﬁer (Altman, 1992) with the image embeddings produced by each expert.
In the case of full experts, we simply apply the corresponding full network to compute these embeddings.
For adapter-based experts, we apply the speciﬁc expert and ignore the remaining ones.
Concretely, let M e ( x ) be the embedding corresponding to expert e on input x , and let D T = { ( x i , y i ) N T i =1 } be our downstream task.
In order to score each expert, we apply a kNN classiﬁer on the embedded dataset D T,e = { ( M e ( x i ) , y i ) N T i =1 } , with k = 1 and Euclidean distance.
The accuracy acc( D T,e ) is computed via leave-one-out cross-validation.
Finally, we select the expert with highest accuracy: e ∗ = arg max e acc( D T,e ) .
There are other alternative proxies that are cheaper than full ﬁne-tuning, for example ﬁtting a logistic regression, SVM, or decision trees to the features.
Although these proxies may better match the ﬁne-tuning accuracy, we use kNN since it is computationally cheap — it only requires a forward pass through the data, and leave-one-out cross-validation requires no additional inference per-fold — and it performs well (section 6).
5. D OWNSTREAM TRANSFER
The expert selection algorithm could choose several experts to be combined to solve any target task.
However, we limit the scope of our work to transferring a single expert per task, since this approach is simple and turns out to be effective.
Thus, the downstream transfer procedure is straightforward: it simply involves ﬁne-tuning the selected expert model.
We ﬁne-tune the entire expert network to the downstream dataset, including the adapters when applicable.
While it was valuable to restrict the scope of upstream training to focus on specializing the expert adapter parameters, we found ﬁne-tuning the whole network downstream to be greatly beneﬁcial.
This differs from the original residual adapters work (Rebufﬁ et al., 2017), where only the adapters were ﬁne-tuned.
E XPERIMENTAL R ESULTS 6. U PSTREAM T RAINING
ImageNet21k (Deng et al., 2009) is a public dataset containing 13 million images, and 14 million labels of 21 843 classes, which are WordNet synsets (Fellbaum, 2012).
In addition to the 21k classes, we consider the 1 741 synsets that are their ancestors.
We use the 50 synsets of ImageNet21k with the largest number of images to train the expert models.
These include e.g. animal, artifact, organism, food, structure, person, vehicle, plan , or instrument .
We released 48 of these ImageNet21k models 2 .
JFT (Sun et al., 2017) is an even larger dataset containing 300 million images and 18 291 classes.
Each image can belong to multiple classes, and as for ImageNet21k, the classes are organized in a hierarchy.
We select as expert domains the classes with a sufﬁciently large number of examples: the 240 classes with more than 850 000 images.
Some of the automatically chosen experts are animal, arts, bird, food, material, person, phenomenon, plant , or product .
We pre-train generic models on a Cloud TPUv3-512, as done in (Kolesnikov et al., 2019).
Then ﬁnetune them brieﬂy on each slice to create the expert models.
Check additional details in appendix D.
6. D OWNSTREAM T ASKS
We evaluate on two suites of tasks, each consisting of several datasets.
The ﬁrst is the Visual Task
Adaptation Benchmark (VTAB) (Zhai et al., 2019), which consists of 19 datasets.
We evaluate on
VTAB-1k, where each task contains only 1k training examples.
The tasks are diverse, and divided into three groups: natural images (single object classiﬁcation), structured tasks (count, estimate distance, etc.), and specialized ones (medical, satellite images).
Appendix E.1 contains further details.
The second suite is a collection of popular natural datasets commonly used in transfer learning literature and, particularly in (Ngiam et al., 2018), to which we compare our work.
6. T RANSFER E VALUATION P ROTOCOL
When transferring to new tasks we need to perform expert selection and choose other hyperparameters (e.g. learning rate for ﬁne-tuning).
For each downstream task, we use the following three step protocol.
Expert Transfer .
We select the expert to transfer using one of the methods presented in section 5.
In both sets of tasks, we use 1k training examples per dataset.
Details are provided in appendix C.1.
Hyperparameter Selection .
In VTAB-1k we use the recommended hyperparameter sweep and 800-training/200-validation split.
We independently repeat the hyperparameter selection procedure 10 times for conﬁdence intervals.
For the other datasets we perform a single random search overhyperparameter sets and select the best set based on the validation performance.
This is a similar computational budget to that of (Ngiam et al., 2018).
See appendices E.2 and F.1 for sweep details.
[Table]
[Table]
Final Re-training .
Using the hyperparameters from the previous step
[Table]
we re-train the selected expert on the entire task (training plus validation set).
In VTAB-1k
[Table]
we repeat this step 3 times for each of the 10 trials of hyperparameter selection and compute the test accuracy, yielding 30 outcomes per method per task.
We compute the median of these 30 outcomes as the ﬁnal accuracy in the dataset.
6. P ERFORMANCE OF D IFFERENT E XPERT S ELECTION S TRATEGIES
We ﬁrst establish which of the expert selection strategies presented in section 5 performs best.
As a baseline we also try selecting a random, uniformly drawn, expert per task.
Table 1 shows the results on VTAB-1k, using full experts trained on JFT.
Table 5 show the results with adapters.
Overall, all methods perform better than random selection, particularly on the N ATURAL group.
This conﬁrms that selecting good experts is essential.
Overall, the performance proxy (kNN) selection performs better than the other alternatives.
kNN’s average accuracy is 11% (relative) and 5.5% higher than that of the domain prediction and label matching, respectively.
Thus, making use of the downstream labels offers a signiﬁcant advantage in expert prediction.
Therefore, in all subsequent experiments we use the kNN-based selection.
We did not see a strong difference for the S TRUCTURED datasets.
We provide an extensive analysis of the kNN accuracy distribution per expert in appendix C.
Appendix G shows how training experts on random subsets of the upstream data does not work well.
6. R ESULTS ON VTAB
Table 2 shows the average accuracy across all the 19 VTAB-1k datasets broken down by type (natural, specialized, and structured).
We summarize our ﬁndings as follows:
Improvement over Non-expert Baseline.
All the algorithms, trained on either JFT or ImageNet21k, improve their corresponding baseline.
Differences are most pronounced on the N ATURAL datasets.
While we also see improvements in S PECIALIZED and S TRUCTURED datasets, some of the conﬁdence intervals overlap.
The performance of both JFT and ImageNet21k models is fairly similar in general.
This is not unexpected; it has been observed before that, with restricted model capacity, they perform very similarly (Kolesnikov et al., 2019).
Appendix C.6 shows the selected experts.
[Table]
Quality of Natural Representations.
The upstream datasets used to train the experts mostly contain natural images.
Consequently, the spectrum of representations offered by our models seem very effective in downstream natural datasets.
More concretely, all models lead to improvements over the baseline performance, with average gains ranging from 1% to over 3.3% on the 7 natural datasets.
Full vs. Adapters.
JFT Experts.
Full models outperform adapters convincingly in N ATURAL and
S PECIALIZED datasets.
However, they do a poor job on S TRUCTURED datasets –mainly due to the failure on one speciﬁc dataset.
ImageNet21k Experts.
In this case, the advantage of full experts comes precisely from S TRUCTURED datasets.
Appendix E provides results broken down by each dataset.
Combining All Experts.
The previous results suggest adding all types of expert models to the selection pool (full or adapter, trained on JFT or ImageNet), since the optimal expert architecture and upstream data vary per task.
Results are remarkable: kNN is able to select good candidates to ﬁne-tune from a pool of almost 600 models, and the relative improvement over the Baseline accuracy across all VTAB datasets is 3.6%, showing gains on all dataset types (see last row in table 2).
6. O UR A PPROACH VS . D OMAIN A DAPTIVE T RANSFER
Domain Adaptive Transfer (Ngiam et al., 2018) (DAT) also relies on specialist models pre-trained on
JFT.
First it trains a generalist model on the upstream data, similar to our B .
For any new task, then re-weights the upstream images based on a forward pass on the downstream data, and ﬁne-tunes a new specialist model using the re-weighted upstream data.
Finally, the model is further tuned on the target task.
DAT falls outside of our transfer setup presented in section 5, as the downstream data directly inﬂuences the upstream training.
This incurs a signiﬁcant cost learning every new target task.
Remarkably, our algorithm works in setups where access to upstream data is not available (e.g. for privacy or proprietary reasons).
We also use downstream labels, which proved to carry key information about the task (see section 6.4).
And most importantly, our method is more practical by amortizing the cost of expert pre-training as more downstream tasks are served.
Under same models and hardware, running kNN (with 240 models) is between 500 × – 1000 × faster than ﬁne-tuning the baseline model with the re-weighted upstream data.
Appendix F includes additional details.
Table 3 shows the mean accuracy over 30 trials per dataset, on the same datasets and under a similar hyperparameter budget as DAT.
These tasks are close to VTAB’s N ATURAL group and yield similar results: full experts outperform adapters.
Our results are not directly comparable to those of DAT since they use Inception-v3 (Szegedy et al., 2016), and AmoebaNet-B (Real et al., 2019) architectures.
Inception-v3 and R50 are similar in performance and size; the former has 24M parameters, attaining 78.8% top-1 on ILSVRC2012 (from-scratch), whereas the latter has 26M parameters and attains 76.0%.
The AmoebaNet-B (N=18, F=512) is 22 times larger, with more than 550M parameters.
Despite the differences, our method is competitive and matches or beats DAT in half the datasets.
D ISCUSSION
Algorithm.
Our results suggest that there are strong potential beneﬁts to using smartly routed pretrained experts when the domain of the experts broadly matches that of the downstream tasks.
We have clearly seen this with natural images.
Instead, as expected, when there is a skill mismatch (e.g. trying to solve a counting task with diverse single-object recognition experts) we have not observed any signiﬁcant gain or loss.
Still, in these cases, the expert selector can fall back on the generic model or representation.
When there is an extremely relevant expert for a task –say, our ﬂower or plant models for the Oxford Flowers 102 task–, using full network experts proved beneﬁcial.
In contrast, many datasets did not have a perfect match, and adapters seemed easier to ﬁne-tune in these cases.
Impact.
In the near future, we foresee large computer vision systems composed by a wide range of pre-trained specialist modules.
These modules may be based on huge amounts of data, small but high-quality curated repositories, or even on private and proprietary content, and they would cover a diverse spectrum of canonical tasks (object recognition, some way of narrow reasoning, counting, sorting, etc.).
Some of them may not even need to be end-to-end learned from data.
Future Directions.
There are a number of exciting follow-up research directions.
Selecting and combining multiple experts for any downstream task is a natural extension of our work.
This could be especially useful for tasks that require understanding several concepts, not necessarily captured by a single expert.
Per-example routing (i.e. applying routes tailored to each individual data-point) could also lead to improvements based on targeted processing, for example, in the context of tasks with instances of various difﬁculties.
Finally, moving beyond our experts based on label hierarchies, and towards automatic discovering and training of experts could unlock even further gains.
A CKNOWLEDGMENTS
We would like to thank Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai and Jessica Yung, who worked on the original BiT, that we use as baseline, and shared their insights with us.
Josip Djolonga and Maxim Neumann provided very useful feedback on an earlier version of this work.
Finally, we thank whole Google Brain team in Zürich for many useful discussions and support.
A E XPERT P REDICTOR N ETWORKS
An expert predictor network (EPN) tries to directly predict the relevant expert for an input image, using only the image itself as input.
We ﬁrst train the EPN upstream, and then we apply it downstream to select the most relevant expert for a new task by aggregating its output on all the images in the task.
EPN Upstream Training.
As we described in section 4, we have split the upstream dataset D U into a collection of subsets { D e : 1 ≤ e ≤ E } , with D e
= ( X e , Y e ) ⊆
D U .
In order to train the
EPN, we simply assign the expert identity e as the label of all images in X e , and train the network in a supervised manner (using softmax cross-entropy) to predict the expert.
Expert slices { D e } are not disjoint, thus, it is possible that an individual image appears multiple times in the training data for the EPN with different expert identities.
Because subsets sizes are different, and in order not to favor any particular expert, we resample the training images so that each expert is seen equally often.
Intuitively, this classiﬁcation problem should be substantially easier than predicting the upstream classes y directly, as there are much fewer experts than upstream classes.
Downstream Expert Selection.
Suppose we are given a downstream task, containing images X T = { x 1 , . . .
, x N T } .
We ﬁrst apply a forward pass on X T using the EPN.
Let Q EPN ( e | X
= x i ) be the probability assigned by the EPN to expert e for input image x i .
In order to make a single decision for the whole downstream task, we combine those probabilities using a log-linear transformation.
We select the expert as follows:
[Equation]
The log-linear combination of per-example probabilities was obtained after several experiments with a number of functions.
Intuitively, this transformation penalizes experts that only apply to a subset of the downstream data, but are not relevant to other downstream examples.
A major drawback of the EPN is the fact that it does not use or beneﬁt from the downstream labels.
Imagine there is an image dataset with pictures containing simultaneously both lions and elephants.
Suppose we are faced with two different downstream tasks based on the same inputs: one is to count lions, the other is to count elephants.
Furthermore, imagine our experts happen to include lion and elephant .
Depending on the task, it would be reasonable to choose one or the other expert.
Unfortunately, the basic EPN approach is agnostic to the outputs, and –as the input images are identical– it would return the same selected expert in both cases.
B K ULLBACK –L EIBLER DIVERGENCE
Since our expert datasets D e were built based on the hierarchy of labels in the upstream dataset, it is reasonable to assume that the prior distribution of the labels in each D e differ across experts e .
Let P e be this prior distribution for expert e .
Then, we can use a divergence measure, such as the
Kullback–Leibler (KL), to determine which expert to use.
If one assumes that the downstream dataset is well represented by the upstream dataset D U (although not necessarily by an individual D e ), one can use the baseline neural network B to approximate the distribution of upstream labels conditioned to the set of downstream images:
[Equation]
where Q B ( Y | X = x j ) is the probability distribution given by B over each image in the set
X T = { x 1 , . . .
, x N T } of downstream images.
Then, we simply select the expert with the lowest KL divergence:
[Equation]
This allows us to leverage the baseline model that we already trained, and not train an auxiliary neural network to predict the expert to use, like the EPN in appendix A does.
In addition, this has the beneﬁt of using information about the distribution of upstream classes, which may be useful when the target classes are well represented among the upstream ones.
In our case, the upstream datasets consists of multi-labeled images.
The distribution of labels given to a particular image is modelled by the neural network as a joint distribution of independent Bernoulli random variables.
Assuming this independence also holds for P e , one can then compute eq.
(3) very efﬁciently.
Of course, this assumption is not true in either case (e.g. the presence/absence of the dog and animal labels is not independent), but it is standard practice for multi-label classiﬁcation.
C F URTHER R ESULTS ON k -N EAREST N EIGHBORS
In this section, we present the kNN accuracy distribution per dataset that we found for both JFT and
ImageNet21k experts.
A ﬂat curve indicates differences across experts may not be very relevant for the downstream task, while steep regions suggest strong decreases in value among expert models.
C. K NN H YPERPARAMETERS
We select the expert to transfer using the kNN transfer proxy with k = 1 and a Euclidean distance metric.
We use 1 000 training examples in all datasets (including those in the comparison with
Domain Adaptive Transfer Ngiam et al.
(2018)), and compute kNN leave-one-out cross-validation to compute the accuracy per expert.
Finally, we select the one with highest accuracy.
For VTAB-1k this corresponds to the entire training set per task; whereas for the other tasks, we randomly sample 1k training examples.
We do not perform special data pre-processing, and simply resize and crop to 224 × 224 , as done in upstream evaluation.
We used a NVDIA V100 GPU to perform the kNN selection for each dataset, with this hardware, selecting among 240 models takes less than 2 hours.
C. A RCHITECTURE C OMPARISONS
In this section we look at the kNN accuracy before transferring the experts, and –in particular– at how it depends on the architecture choice.
Recall each expert is associated with one slice of the upstream data.
For any given slice, we have trained both a full ResNet50 network, and adapters attached to a pretrained ResNet50.
We plot the accuracy achieved by these representations (scatter-plot, full at x -axis; adapters at y -axis) for all experts for each group of VTAB datasets.
We look at JFT experts (Figure 4) and at ImageNet21k experts (Figure 5).
Ideally, we would expect some positive correlation if expert representations were somewhat similar regardless of the architecture.
JFT Experts.
The ﬁrst seven plots in Figure 4 show the results in natural datasets.
While most experts do not seem relevant –and performance seems a bit uncorrelated between both types of models–, in most datasets we see that there are a few good experts (top right corner) which offer the strongest performance despite of the selected architecture.
SVHN seems to be an exception.
Similar plots are displayed in the following 4 and the last 8 plots in Figure 4 for specialized and structured datasets, respectively.
Few datasets show agreement on the most promising expert slices, such as Eurosat or Resisc45.
Unfortunately, there is no clear agreement in most specialized and structured datasets.
ImageNet21k Experts.
We see a reasonable agreement among the best ImageNet21k experts in natural datasets (see ﬁrst 7 plots in Figure 5).
While not as correlated as in the case of natural datasets, we still see some positive relationship in some specialized (Eurosat, Resisc45, Patch Camelyon) and structured (Clevr Count, DSprites Position, Smallnorb Azimuth) datasets.
C. K NN A CCURACY D ISTRIBUTION FOR JFT E XPERTS
Figure 6 shows the distribution of the kNN accuracy obtained from the embedding of each of the experts trained on JFT.
In each case (full and adapters), the kNN accuracy of the 240 experts has been sorted in a decreasing manner.
Note that we pick the single expert with highest-score (although other approaches are possible).
[Figure] [Figure]
[Figure]
In most • N ATURAL datasets, we observe that full JFT experts are on average better than their adapter counter-parts.
In datasets like Caltech101, Cifar100, or DTD, it seems these differences do not affect the top experts, while in others (such as Flowers, Pets, and Sun397)
the differences still apply to the best expert.
Also, overall, we see that in natural datasets there are usually strong differences between good and bad experts.
The range of kNN accuracies is pretty large for Caltech101, Flowers, or Pets, where some experts seem to already solve the task, while others lead to quite poor accuracies.
The latter may be ﬁxed to some extent by downstream ﬁne-tuning.
In the • S PECIALIZED group we see a similar pattern in the comparison between full and adapterbased experts.
However, the accuracy range of variations (except, maybe, at the very worst end) is narrower.
The story for • S TRUCTURED datasets with JFT experts is a bit different.
In some datasets, adapters models lead on average to better initial representations (such as Clevr Closest and Clevr Count, or dSprites Position).
As with structured datasets, the difference between the best and the worst experts is shorter.
This may be in part explained by the hardness of the task itself (the average accuracy after
[Table]
A d a p t e r s k NN
A c u r a c y
[Table]
A d a p t e r s k NN
A
[Table]
r a
[Table]
y
[Table]
NN
[Table]
A d a p t e r s k NN
A c u r a c y
[Table]
[Figure]
ﬁne-tuning is deﬁnitely lower than in the natural case), but there are some counter-examples to this, like dSprites Position where ﬁnal accuracies go up to around 90%.
[Figure]
[Table]
Figure 7 presents the distribution of the kNN accuracy obtained from the embedding of each of the experts trained on the ImageNet21k dataset.
For context, in all the plots we also show the Top-50 JFT full experts.
For • N ATURAL tasks, we observe that the quality of the ImageNet21k expert representations is way more homogeneous than the JFT one.
Accordingly, ﬁnding the right expert in JFT may be more important (as accuracy decreases fast), while it may provide even more target-tailored representations (see Oxford Flowers and Oxford Pets).
Overall, ImageNet21k accuracies seem more stable, and differences between full and adapters are modest.
Overall, both full and adapter ImageNet21k experts seem to perform similarly on • S PECIALIZED tasks.
The plots suggest that ImageNet21k experts are a bit ahead of the full JFT ones (even though the gap at the top tends to close).
We see a few distinct behaviors in • S TRUCTURED .
There tend not to be very remarkable winner experts, and full experts may provide a small boost compared to adapter-based ones.
In most datasets, the Top-50 full JFT experts outperform the ImageNet21k ones.
[Figure]
C. K NN A CCURACY D ISTRIBUTION FOR C ONSECUTIVE C HECKPOINTS OF I MAGE N ET 21 K B ASELINE
In this subsection
[Figure]
we study how representations evolve during training.
In order to do that, we stored 157 checkpoints –equally spaced– over the training of our ImageNet21k baseline.
We trained the model for 90 epochs.
For each dataset, we compute the kNN accuracy of the checkpoints, and display the curves in ﬁg. 8.
As an auxiliary line, we also show the mean kNN accuracy across all the checkpoints.
There are some clear differences depending on the type of dataset.
In the case of • N ATURAL images, it seems that more training leads to better representations.
The kNN accuracy tends to increase (Cifar100 and SVHN are exceptions).
• S PECIALIZED datasets behave in a different way; while there is an initial boost in accuracy (i
[Figure]
e. trained models are better than randomly initialized ones), long training only leads to very minor improvements in representation quality for these tasks.
Finally, •
S TRUCTURED datasets have extremely ﬂat footprints.
This probably means that our semantic experts are not a good ﬁt for this type of task.
[Figure]
C. S ELECTED E XPERTS
The following table presents the experts selected by kNN in each of the individual datasets of the •
N ATURAL , • S PECIALIZED and • S TRUCTURED groups.
[Table] [Table]
• Cifar
[Table]
• DTD
[Table]
• Oxford Flowers
[Table]
• Oxford Pets
[Table]
• Sun
[Table]
• SVHN
[Table]
• Diabetic Retinopathy
[Table]
• Eurosat
[Table]
• Patch Camelyon
[Table]
• Resisc
[Table]
• Clevr Closest
[Table]
• Clevr Count
[Table]
• DMLab
[Table]
equipment
[Table]
• dSprites Orientation
[Table]
plant
[Table]
• dSprites Position
[Table]
• Kitti
[Table]
• Smallnorb Azimuth
[Table]
• Smallnorb Elevation
[Table]
D U PSTREAM TRAINING D. U PSTREAM T RAINING D ETAILS
Unconditional pre-training .
We pre-train generic JFT and ImageNet21k models using a similar protocol to the one described in Kolesnikov et al.
In particular, we use SGD with momentum of 0.9, with a batch size of 4096, an initial learning rate of 0.03 (scaled by a factor of batch size 256 ), and weight decay of 0.001.
The JFT backbone model is trained for a total of 30 epochs, while the
ImageNet21k is trained for 90 epochs.
In both cases we perform training warm-up during the ﬁrst 5 000 steps by linearly increasing learning rate, and then decay the learning rate by a factor of 10 at { 13 , 23 , 56 } of the total duration.
During this phase we used a Cloud TPUv3-512 to train each of the baseline models, which takes about 25 hours in the case of JFT and .
Experts training .
In order to obtain the expert models, we then further tune these baselines (adding the residual adapters, when applicable) on different subsets of the original upstream dataset.
We use a similar setting to the one described before, although we train for much shorter times, use a batch size of 1 024, and use different learning rates.
In particular, the full experts use an initial learning rate 10 − 4 , since all the parameters were pre-trained in the earlier phase.
The experts with adapters use a larger learning rate of 10 − 1 , as these components are trained from scratch and are the only ones that are tuned.
We use the same learning scaling factor and decay schedule as in the previous step.
The initial learning rate in each case was decided based on average upstream performance across the different expert datasets.
We ﬁne-tune the full experts for 2 epochs, and the adapters forepochs, relative to the size of the entire dataset.
The only exception is for the results reported in the comparison with Ngiam et al.
(2018), for which we observed in the validation data that full experts trained for 4 epochs performed better.
In both stages, we perform standard data augmentation during training, which includes random image cropping as in Szegedy et al.
(2015), random horizontal mirroring, and ﬁnally image resize to a ﬁxed resolution of 224 × 224 .
When we need to evaluate these models on upstream data (i.e. upstream learning rate selection), we simply resize and crop the images to a ﬁxed resolution of 224 × 224 .
Pixel values are converted to the [ − 1 , 1] range.
During this phase we used a Cloud TPUv3-32 to train each of the experts.
Training one of the JFT experts for 2 epochs takes about 11 hours, while this is reduced to 30 minutes in the case of the
ImageNet21k experts.
D. U PSTREAM F REEZING
Note that many expert datasets D e do not contain any instances of some original upstream classes (for example, the data for the expert elephant may not contain any image with the label vehicle ).
As the head is frozen and shared among all experts, the adapters need to ﬁnd other ways to ignore classes that do not apply at all to the expert.
We found this to be beneﬁcial in practice, as we avoided too much upstream dependence on the head (which is later discarded in the transfer stage).
E V ISUAL T ASK A DAPTATION B ENCHMARK DETAILS E. C LASSES PER T ASK
The number of classes in the VTAB tasks varies signiﬁcantly, see Figure 9.
As we are most interested in the low-data regime, we ﬁx the number of downstream examples to 1 000, implying that some downstream datasets only contain 3-10 examples per class –like Sun397 or Caltech.
E. D OWNSTREAM
We use SGD with momentum of 0.9 and a batch size of 512.
The initial learning rate is scaled by batch size 256 .
We don’t perform any data augmentation, and resize all the images to a ﬁxed resolution of 224 × 224 .
Pixel values are converted to the [ − 1 , 1] range.
We perform a restricted hyperparameter search, in particular we follow the lightweight suggestion from Zhai et al.
The sweep tries in total 4 different sets of hyperparameters for each dataset.
• Initial learning rate: the values of { 0 . 1 , 0 . 01 } .
• Training schedule: we try with a total training duration of 2 500 , 10 000 steps, with a linear warm up of the scaling rate of 200 and 500 steps, respectively.
For both durations, the learning rate is reduced by a factor of
Note that the hyperparameter sweep is done by training the models on 800 examples (out of the 1available), and selecting over 200 validation examples.
Then, the best combination of hyperparameters is used to re-train the models on the full 1 000 data points.
Because the variability due to random initialization with so few data points is large in some datasets, we perform 10 independent runs of hyperparameter selection, and then re-training the modelstimes for each of the 10 selected hyperparameters.
This yields a total of 30 outcomes for each of the VTAB-1k datasets.
For each dataset, we report the median over these 30 trials and compute (percentile) bootstrapped conﬁdence intervals at 95% level.
For downstream training we use a Cloud TPUv3-16.
In VTAB-1k, the running time depends on the number of steps.
It takes 12 minutes to ﬁne-tune one of our experts for 10 000 steps (the longest duration), and just 3 minutes for the shorter schedule of 2 500 steps.
E. A DDITIONAL R ESULTS OF D IFFERENT E XPERT S ELECTION S TRATEGIES
In section 6.4, and more precisely in table 1, we studied the performance of random transfer (expert selection uniformly at random).
There, we only reported the results corresponding to full experts trained on JFT.
For completeness, table 5 shows also the results with adapters.
The pattern is fairly similar: random transfer leads to massive losses in Natural datasets (we obtain an almost 35% improvement by applying kNN with respect to random transfer).
Domain Prediction and Label
Matching also heavily help in these settings.
In Specialized and Structured tasks, both Domain
Prediction and Label Matching seem to offer little-to-no gains, whereas kNN still leads to a modest boost on Structured, and a decent one on Specialized.
Overall (last column of the table), the improvement is signiﬁcant and strong for all routing methods.
[Table]
E. P ER -T ASK R ESULTS
All VTAB results presented so far were averaged over dataset types (natural, specialized, and structured).
In this subsection, we break down the outcomes per dataset.
Table 6 shows the mean accuracy (and conﬁdence intervals) for 13 algorithms and 19 datasets.
The datasets are sorted according to the data type.
The table can be used for reference.
Some datasets showcase a wide range of outcomes for any ﬁxed algorithm.
In order to expose this in a clear fashion, we present in ﬁg.the individual trial accuracies for the best algorithms and baselines, in all of the VTAB datasets.
The ﬁne-tuning process on datasets like Clver Count, dSprites xPosition, or SVHN deﬁnitely shows a high variance of test accuracies.
The median estimator partially mitigates this effect.
[Table] [Table]
JFT + IN21k
[Table]
c a l t e c h c a m e l y o n c i f a r c l e v r .
c
l o s e s t c l e v r .
c o un t d
m l a b d s p r
i t e s .
o r i e n t d s p r
i t e s .
x p o s d t d e u r o s a t f l o w e r s k
i tt i p e t s r
e
s i s c r e t
i n o s m a ll n o r b .
a z m t h s m a ll n o r b .
e l e v s un s v hn 0.
A cc u r a c y
Baseline (JFT)
Baseline (IN21k)
All Experts (JFT)
All Experts (IN21k)
All Experts (JFT + IN21k)
F D ETAILS ON THE C OMPARISON WITH D OMAIN A DAPTIVE T RANSFER F. H YPERPARAMETERS
We randomly explored the space of hyperparameters drawing 36 samples from the following distributions:
• Initial learning rate: Log-uniform in [2 · 10 − 4 , 2 · 10 − 1 ] .
• Total training steps: Uniform in { 2000 , 4000 , 8000 , 16000 , 32000 } .
• Weight decay: Log-uniform in [10 − 6 , 10 − 2 ] .
• Mixup α : Uniform in { 0 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 } .
We use a batch size equal to 512 and decay the learning rate by 0.1 at 30%, 60% and 90% of the training duration.
During the ﬁrst 10% of training steps, we linearly warm up the learning rate.
Standard data augmentation techniques are applied to prevent overﬁtting.
During training, for all datasets except CIFAR10 (which has a smaller resolution), we resized the images to a ﬁxed size of 512 pixels on both sides, then randomly cropped it to a patch of 480 pixels, randomly ﬂipped the image horizontally, and converted the pixel values to the [ − 1 , 1] range.
For CIFAR10, we used a resolution of 160 pixels during the resize and crops of 128 pixels.
During evaluation, we simply resize the images and convert the pixel values analogously.
We ﬁnd the best hyperparameter for each dataset based on the accuracy on the validation data, and then, applying the corresponding set of hyperparameters, the selected expert is ﬁne-tuned again on the union of the training and validation examples of the dataset.
The accuracy on a test set is reported.
We re-trained the models 30 times using different random seeds.
For downstream training we use a Cloud TPUv3-16.
In DAT, the running time depends on the number of steps selected as the hyperparameter and the resolution of the images.
At most, it takes 150 minutes to ﬁne-tune one of our experts on one downstream dataset, and at least it takes 8 minutes.
F. D ETAILED RESULTS
Table 7 shows the mean accuracy across those 30 trials and the 95% bootstrapped conﬁdence intervals, for each of our experts and selection algorithms, for each dataset as well as the average across the six datasets
. F. D IFFERENCES IN ASYMPTOTIC RUNNING TIME
Table 8 contains an asymptotic analysis of the different phases of each approach.
In our case, upstream training includes both the cost of training the generic backbone network for S U steps, and the cost of
[Table]
[Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
training each of the E experts for S A steps, with a batch size of B .
In the case of Domain Adaptive
Transfer (DAT), only the ﬁrst cost is incurred in this phase.
Observe that in our case the cost of upstream training is amortized over the number of tasks that one has to learn, since it’s only incurred once.
In the downstream phase, both methods need a forward pass over the number of downstream examples,
N T .
Then, leaving the number of parameters of the model aside, the cost of Ngiam et al.
(2018) is dominated by S A · B , which is the total number of examples used to ﬁne-tune the baseline model to a weighted/resampled version of the upstream data, and ours is dominated by N T · E , the cost of running a forward pass of the downstream data in each of the pre-trained experts.
In practice, because
S A · B (roughly 1 . 2 · 10 9 , when ﬁne-tuning for 4 epochs on JFT) is much larger than N T · E (roughly 5 · 10 5 , when using 1 000 downstream examples for selecting over 500 experts), our approach is much faster.
Thus, our approach should be roughly three orders of magnitude faster than that of Ngiam et al.
(2018), when learning a new task.
The ﬁnal cost of ﬁne-tuning to the downstream dataset is the same in both cases, and it’s negligible in comparison.
As described in appendix C.1, we use a single NVIDIA V100 GPU as the hardware accelerator when selecting the expert model to use for a given downstream task.
It requires less than 2 hours 3 to select a single expert model among a pool of 240 experts, based on 1 000 images from the downstream task.
This is the real duration of the “Expert preparation” step, following our method, depicted in table 8.
Under the same hardware constraints, ﬁne-tuning a R50 on JFT for 4 epochs, using the details described in appendix D, requires around 1780 hours.
This is essentially the duration of the “Expert prepartion” step, following the Domain Adaptive Transfer approach 4 .
Given that ﬁne-tuning on the downstream dataset takes only a few minutes (see appendix E.2), our transfer approach is 890 × faster than that of Domain Adaptive Transfer.
Depending on different optimizations and bottlenecks, we expect our approach to be 500 × − 1000 × faster in other scenarios.
G T HE V ALUE OF S EMANTIC E XPERTS
In section 6.4, we have seen that a smart choice of experts leads to substantial gains with respect to a single model trained on all the upstream data.
Here, we rule out the possibility that these gains come merely from the fact that we are able to select a representation among a wide range of choices by directly testing their initial predictive power on the downstream task.
To do so, instead of training our experts in subsets of JFT based on its label hierarchy, we fully ﬁne-tuned the baseline model onuniformly random subsets of JFT, with sizes matching the size of our original semantic experts.
We did this independently with both adapter and full experts.
Then, we applied kNN to select the best random expert on each downstream dataset.
Note this is not at all equivalent to applying transfer at random as in section 6.4.
In principle, it was not even clear if this approach with random experts would outperform the baseline.
Figure 11 (a) and (b) show our results for adapter and full experts respectively.
In both cases, the overall performance drops when we replace semantic experts with random ones.
This difference seems stronger in the case of adapter-based experts.
However, notice that the full expert results are very inﬂuenced by strong negative results in one of the structured datasets (Clevr
Count), as shown in table 6.
Also, random experts results are comparable to the baseline.
Note that random experts are not dumb; they are just a diverse set of models with the general ﬂavor of the upstream dataset.
The algorithm can still beneﬁt from their diversity when confronted with a new task, whereas we expect them to be more similar to each other than in the semantic case.
The semantic experts are trained mostly on natural slices, and we see a large improvement in the
N ATURAL tasks when we use them (2.7% and 4.7% gains for adapters and full, respectively, compared to random experts).
This reinforces the idea that experts in the right domain can be very helpful.
Moreover, in N ATURAL tasks, the baseline outperforms random experts; this suggests that here more data is better unless data is smartly selected.
As we have hypothesized before, it seems that our natural-image experts do not provide a meaningful expertise or competitive edge on S TRUCTURED tasks.
We see a large improvement on S PECIALIZED tasks when using full experts, while the effect is not there for adapters.
Accordingly, we would not read too much into these results.
A v e r a g e a cc u r a c y 69.
All VTAB      77.
Natural       81.
Specialized      57.
Structured
Baseline
Random
Semantic (a) JFT Adapter Experts
A v e r a g e a cc u r a c y 69.
All VTAB       77.
Natural      81.
Specialized      57.
Structured
Baseline
Random
Semantic (b) JFT Full Experts