Meta-Learning via Classifier(-free) Guidance
Abstract
We aim to develop meta-learning techniques that achieve higher zero-shot performance than the state of the art on unseen tasks.
To do so, we take inspiration from recent advances in generative modeling and language-conditioned image synthesis to propose meta-learning techniques that use natural language guidance for zeroshot task adaptation.
We first train an unconditional generative hypernetwork model to produce neural network weights; then we train a second “guidance” model that, given a natural language task description, traverses the hypernetwork latent space to find high-performance task-adapted weights in a zero-shot manner.
We explore two alternative approaches for latent space guidance: “HyperCLIP”-based classifier guidance and a conditional Hypernetwork Latent Diffusion Model (“HyperLDM”), which we show to benefit from the classifier-free guidance technique common in image generation.
Finally, we demonstrate that our approaches outperform existing meta-learning methods with zero-shot learning experiments on our Meta-VQA dataset.
Introduction
State-of-the-art machine learning algorithms often lack the ability to quickly generalize in a sample efficient manner to new unseen tasks.
In contrast, humans show remarkable capabilities in leveraging previous knowledge for learning a new task from just a few examples.
Often, not even a single example is needed, as all relevant task information can be conveyed in the form of natural language instructions.
Inspired by recent advances in conditional image generation (Ramesh et al., 2022; Rombach et al., 2022), we reframe meta-learning as a multi-modal generative modeling problem such that, given a task, its adapted neural network weights and its natural language description are considered equivalent multi-modal task representations.
We show that popular techniques for the image domain, such as
CLIP-based guidance (Gal et al., 2021; Patashnik et al., 2021), denoising diffusion models (Ho et al., 2020), and classifier-free guidance (Dhariwal and Nichol, 2021; Ho and Salimans, 2021; Nichol et al., 2022) can be repurposed for the meta-learning setting to generate adapted neural network weights instead of images.
We point to Appendix A.1 for other related work.
Specifically, we approach the generation of neural network weights in two separate phases.
In the unconditional pre-training phase we train a generative hypernetwork (Ha et al., 2016) to map from its latent space to the weight space of a base model (Figure 1.A).
In the guidance phase, we learn language-conditioned models that can be used to traverse the hypernetwork latent space and find zero-shot adapted weights with high performance on our task (Figure 1.B and 1.C).
Meta-Learning with Multi-Modal Task Embeddings
The setting we investigate is similar to the classic meta-learning framework, where we operate within a distribution of tasks
T i ∼ p
( T ) , each associated with a loss function L T i .
Using a set of training tasks drawn from this distribution, our goal is to train a model f
( x, W i ) =
y such that it generally performs well on new unseen tasks drawn from p ( T ) .
In this work, we assume to have access to a natural language description t
i for each task T i , which can be encoded into a task embedding e i using a pre-trained language model.
The fundamental building block of our unconditional generative model is the hypernetwork (Ha et al., 2016)
h ( z, θ ) =
W , which can be trained either with classic MAML or with a VAE setup, paired to an encoder d ( W, ω ) =
( µ z , Σ z ) .
For more details, we point to Appendix A.3.
HyperCLIP: Training a CLIP Encoder for the “Meta-Learning Modality”
Our first approach builds on top of Contrastive Language-Image
Pre-training (CLIP) (Radford et al., 2021).
In the original CLIP formulation, separate text and image encoders are trained such that, given a bi-modal sample ( x i , t i ) of an image and its corresponding language caption, their representations (CLIP I ( x i ) =
e ( I ) i and CLIP T ( t i ) =
e ( T ) i ) are aligned across modalities.
In our work, we consider multi-modal representations of meta-learning tasks T i , which may be presented in the form of language as task descriptions t i , but potentially also in the form of images, videos, and audio.
We fine-tune a base machine learning model f
( x, W i ) =
y for task T i and consider the base model as part of an alternative meta-learning modality for task T i .
The meta-learning modality can then be paired in contrastive learning with the other multi-modal descriptions of T i .
We can thus define our new HyperCLIP encoder as a “reverse hypernetwork” CLIP H ( W i ) =
e ( H ) i .
We then introduce HyperCLIP guidance , the first algorithm for classifier guidance in the metalearning setting (Figure 1.B).
Given a previously unseen validation task T i and an unconditional generative hypernetwork model h ( z, θ )
=
W , we can use a trained HyperCLIP encoder
B A Who is wearingglasses?
CLIP T
CLIP H
y
ij f w x ij
T
T N
H
H N ...
CLIP T
T 1 ,T
T N
H l CLIP H y ij f w x ij max.
T
T N T 2 C max.
How is the weather?
Who is wearing glasses?
Where is the child sitting?
to the latent space of a pre-trained language encoder CLIP T , which we use to embed the natural language questions associated to the tasks (see A ).
To perform task inference given an already finetuned network, we encode all candidate task questions using the language CLIP encoder (see B ), then encode the fine-tuned network weights with HyperCLIP (see C ), and finally infer the correct task with a softmax operation over cosine similarities between HyperCLIP and language CLIP embeddings. CLIP H ( W ) =
e ( H ) to guide the exploration of the hypernetwork’s latent space and find a set of base weights W i with high zero-shot performance for T i . Specifically, as long as we are given a starting hypernetwork latent vector z 0 and a textual description t i of the task, we can update z 0 with gradient descent over the guidance loss
[Equation]
[Equation]
and then run the optimized latent vectors ˆ z i through the generative hypernetwork to find adapted zero-shot base network weights h (ˆ z i , θ )
=
ˆ W i that perform well for the task.
HyperLDM: Task-conditional Diffusion of Hypernetwork Latents
Recent advances in image generation involve the use of classifier guidance and classifier-free guidance during the sampling process of a Diffusion Model (Dhariwal and Nichol, 2021;
Ho and Salimans, 2021; Kim et al., 2022; Crowson, 2022; Nichol et al., 2022; Rombach et al., 2022).
To paint a more complete picture, we also investigate this setting in the meta-learning domain.
In our meta-learning setting, we aim to train a diffusion model which generates adapted zero-shot base network weights
ˆ W i that perform well for task T i .
Thus, our diffusion model has to be conditional on a task embedding e i .
Moreover, in order to speed up training and leverage our previously trained generative hypernetwork h ( z, ψ ) , we define the diffusion process on latent vectors instead of doing so in weight space, emulating the Latent Diffusion technique from Rombach et al.
We propose
Hypernetwork Latent Diffusion Models ( HyperLDM ), which learn to sample from the conditional distribution of fine-tuned latent vectors p ( z 0 | e i ) given a language CLIP embedding corresponding to the task.
The HyperLDM neural network models the noise function ϵ ψ ( z t , t, e i ) , and is learned by optimizing the reweighted variational lower bound, which in this setting is
[Equation]
The classifier guidance technique presented in Section 3 can be also adopted together with diffusion models.
Even in the case of conditional diffusion models, the gradient of an auxiliary classifier (or
CLIP encoder) can be added during sampling to induce an effect similar to GAN truncation (Brock et al., 2018), producing samples that are less diverse but of higher quality.
The classifier-free guidance technique (Ho and Salimans, 2021; Nichol et al., 2022) allows us to leverage a conditional diffusion model to perform the same tempered sampling as above, without the auxiliary classifier.
To do so, we train the conditional network ϵ ψ ( z t , t, e i ) to also model the unconditional case ϵ ψ ( z t , t ) .
One way of doing this is with conditioning dropout , simply dropping the conditional input e i for a certain percentage of training samples, inputting zeros instead.
We can then sample at each diffusion iteration with
[Equation]
For γ = 0 , this recovers the unconditional diffusion model, while for γ = 1 it recovers the standard task-conditional model.
For γ > 1 , we instead obtain the classifier-free guidance effect, which we show results in the sampling of latent vectors ˆ z i corresponding to higher-performing task-conditional network weights h (ˆ z i , ψ ) =
ˆ W i .
We point to a more in-depth discussion on classifier-free guidance and its connection to score matching in Appendix A.6.
Experimental Setup and Results
In this section, we demonstrate the soundness of our two approaches with zero-shot and few-shot image classification experiments against a series of traditional meta-learning baseline techniques.
Throughout our experiments, we fix the choice of base network model (see Appendix A.7), only varying the meta-learning techniques employed to obtain adapted base model weights.
[Figure]
u e r y
S e t A cc u r ac y [Figure] [Figure] In Table 1 from Appendix A.9 we show how well our methods fare on the Meta-VQA dataset (see
Appendix A.8 for discussion on the dataset).
We test HyperCLIP Guidance and HyperLDM when trained on top of either a hypernetwork or a VAE generator (see Appendix A.7 and A.11 for more details).
HyperCLIP Guidance allows for faster sampling than HyperLDM but is generally less performant (53.98% acc.)
, still, it improves upon all other zero-shot baselines except for conditional multi-task learning ( Cond.
Multitask , 54.12%).
The best performing model for the zero-shot setting is HVAE + HyperLDM , with an accuracy of 54.57%, and specifically for classifier-free guidance with γ = 1 . 5 .
As illustrated in Figure 3.a, to further show the effectiveness of the classifier-free guidance technique, we sweep over several candidate γ parameters to find that the optimum occurs for γ > 1 .
In another experiment, as shown in Figure 3.b, we train our model while only keeping 50% or 10% of task descriptors, and show that while this strongly impacts traditional Cond.
Multitask learning, it does not affect HyperLDM as strongly due to its two-phased training regime based on an uncondtional VAE.
In fact, even when dropping half of task descriptors, HyperLDM still performs better than the Uncond.
Multitask baseline.
Conclusion
In this work we introduced a framework that re-interprets meta-learning as a multi-modal generative modeling problem.
Our HyperCLIP guidance and HyperLDM methods leverage this insight to generate task-adapted neural network weights in a zero-shot manner given natural language instructions, and constitute the first application of the CLIP guidance and classifier-free guidance techniques from image generation to the meta-learning domain.
Our experiments show that our methods successfully make use of external task descriptors to produce high-performance adapted networks in the zero-shot setting.
Acknowledgments
We are grateful for funding received by the ETH AI Center, Swiss National Science Foundation (B.F.G. CRSII5-173721 and 315230 189251), ETH project funding (B.F.G. ETH-20 19-01), the Human Frontiers Science Program (RGY0072/2019), and Credit Suisse Asset Management. Code Release
Our code is available at https://github.com/elvisnava/hyperclip . Author Contributions Elvis Nava * Original idea, implementation and experiments (guidance techniques and diffusion), paper writing (overall manuscript structure, generative modeling and guidance sections, experimental results section). Seijin Kobayashi Implementation and experiments (maml and hypernetworks, hyperclip guidance), paper writing (meta-learning background and related work).
Yifei Yin Creation of the Meta-VQA dataset, initial experiments on the CLIP-Adapter base model.
Robert K. Katzschmann Conceptualising of the project idea (focus on future robotics applications), participating in project meetings, giving feedback and conceptually guiding the approaches, giving feedback to the manuscript. Benjamin F. Grewe Conceptualisation of the project with a focus on using language task descriptors, senior project lead, participating in project meetings, providing conceptual input for developing the two approaches, giving feedback on the manuscript, developing intuitive schematics for Figureand 2. A Appendix A.1 Related Work Hypernetworks By introducing multiplicative interactions within neural networks (Jayakumar et al., 2019), hypernetworks (Ha et al., 2016) have been shown to allow the modeling of diverse target network weights in, e g. , continual learning, even in the compressive regime (von Oswald et al., 2021a, 2020) without loss of performance.
For a given supervised problem, hypernetworks have been used to model the complex Bayesian posterior of the weights in conjunction with variational inference (Henning et al., 2018; Krueger et al., 2018).
Similar approaches have been used for continual learning, whereby task-specific weight posteriors are generated by a task-specific hypernetwork, itself generated by hyper-hypernetwork conditioned on a learned task embedding (Henning et al., 2021).
Another vein of work consists in using hypernetworks to distill ensembles of diverse networks (Wang et al., 2018;
Ratzlaff and Fuxin, 2020; von Oswald et al., 2021a).
Meta learning In the context of meta-learning, hypernetworks have been successfully used in combination with popular gradient-based meta-learning methods (Finn et al., 2017), especially benefiting in few-shot classification and regression problems (Zintgraf et al., 2019;
Zhao et al., 2020;
Flennerhag et al., 2020).
More generally, related works have shown the usefulness of learning a low dimensional manifold in which to perform task-specific gradient-based adaptation at meta test time (Rusu et al., 2018;
von Oswald et al., 2021b; Lee and Choi, 2018), instead of directly adapting in weight space.
Recent works bypasses the formal bi-level formulation of meta-learning by, e.g. , using transformers to directly output the weights of the target network using both the few-shot labeled data and transductive data as input (Zhmoginov et al., 2022).
Generative Modeling and Classifier(-free) guidance A plethora of techniques have been proposed for the generation of samples from high-dimensional domains such as images, such as Generative Adversarial Networks (Goodfellow et al., 2014; Brock et al., 2018, GANs) and Variational Autoencoders (Kingma and Welling, 2014, VAEs).
Denoising Diffusion Probabilistic Models (Sohl-Dickstein et al., 2015; Ho et al., 2020, DDPM) overcome common issues in generative modeling using a simple likelihood-based reconstruction loss for iterative denoising, and have been shown to achieve state-of-the-art results in high resolution image generation (Dhariwal and Nichol, 2021; Rombach et al., 2022).
Several techniques have been proposed for effective conditional sampling in generative and diffusion models, such as classifier/CLIP guidance (Dhariwal and Nichol, 2021; Gal et al., 2021;
Patashnik et al., 2021) and classifier-free guidance (Ho and Salimans, 2021; Crowson, 2022; Nichol et al., 2022).
Diffusion models with classifier-free guidance have also been successfully applied in non-visual domains, such as audio generation (Kim et al., 2022) and robotic planning (Janner et al., 2022).
A.2 Model-Agnostic Meta-Learning
We present here a slightly altered formulation of MAML (Finn et al., 2017) introduced in (Zintgraf et al., 2019), whereby the parameters of the model g are partitioned into two parts: context parameters ϕ that are adapted on individual tasks, and shared parameters θ that are meta-trained and shared across tasks.
MAML and its variants focus on the few shot setting, which aims to learn an initialization for these parameters such that the model g ( · , θ, ϕ ) generalizes well on new tasks after fine-tuning ϕ on a few data points from that task.
To train such a model, the data from each task T i is split during training into a support set D s i and a query set D q i .
The MAML objective aims to optimize the validation score evaluated on the query set when fine-tuning ϕ on the support set, e.g. , consider the following optimization problem:
[Equation]
where A T i is some differentiable algorithm, typically implementing a variant of few-step gradient descent on the loss computed on the support set, e.g. , in the case of one-step gradient descent:
[Equation]
with some learning rate η .
The objective from Eq. 4 is itself solved with gradient descent, by iteratively optimizing the parameters ϕ in the inner loop on the support set of a sampled task, and updating θ and the initialization of ϕ with their gradient with respect to the entire inner loop training process, averaged over batches of tasks.
Note that the original formulation of MAML considers θ = ∅ .
If we consider hypernetworks, We can rewrite the MAML objective with respect to the hypernetwork weight θ as [Equation]  
[Equation]
(6) when A T i ( D s i , e i , θ )
=
e i , we recover the classic multi-task objective of a hypernetwork optimizing for zero-shot performance.
When A T i is instead the gradient descent algorithm on e i , the objective aligns with the few-shot performance of h when adapting the embedding initialized at e i .
A.3 Hypernetworks as Generative Models of Network Weights
We hereby define our two alternative choices for hypernetworks as unconditional generative models of base neural network weights: 1) We define a Hypernetwork VAE as in Figure 1.A, which, given samples of fine-tuned base network weights W i , learns a low-dimensional normally distributed latent representation z .
The encoder d ( W, ω ) =
( µ z , Σ z ) maps base network weights to means and variances used to sample a latent vector z , while the decoder (or generator) is a classic hypernetwork h ( z, θ )
=
W which reconstructs the network weights from the latent vector. 2)
Using MAML, we learn both an embedding z and hypernetwork weights θ such that, when fine-tuning only the embedding z on each task T
i , we obtain high-performing base networks with weights W i
= h ( z i , θ ) .
Concretely, we optimize θ and the initialization of z following the objective in Eq.
4 where z takes the role of the task-specific parameter ϕ .
A.4 The HyperCLIP Training Algorithm
Here we report the detailed learning procedure for training the HyperCLIP encoder.
Algorithm sample a batch of tasks T i =1 ,...,N with loss functions L T i , training data D train i and text t i define two N -sized arrays of d -dimensional embeddings T ∈ R N × d and H ∈ R N × d for i = 1 , . . .
, N do
T [ i ] =
CLIP T ( t i ) /
∥ CLIP T ( t i ) ∥
Fine-tune W i with objective: min W (cid:80) ( x ′ ,y ′ )
∈ D train i L T i ( f ( x ′ , W ) , y ′ )
H [ i ] =
CLIP H ( W i ) / ∥
CLIP H ( W i ) ∥ end for loss =
(cid:0) L cross-entropy ( TH ⊤ )
+ L cross-entropy ( HT ⊤ ) (cid:1) /
Update weights of CLIP H ( . ) using ∇ loss
A.5 Diffusion Models
Denoising Diffusion Probabilistic Models (Sohl-Dickstein et al., 2015; Ho et al., 2020, DDPM) are a powerful class of generative models designed to learn a data distribution p ( x ) .
They do so by learning the inverse of a forward diffusion process in which samples x 0 of our data distribution are slowly corrupted with additive Gaussian noise over T steps with a variance schedule β 1 , . .
. , β T , resulting in the Markov Chain
[Equation]
A property of such a process is that we can directly sample each intermediate step from x 0 as x t =
√ ¯ α t x 0 + ( √ 1 −
¯ α t )
ϵ given ϵ ∼ N (0 , I ) , α t = 1 − β t and ¯ α t = (cid:81) ts =1 α t .
Then, to learn the reverse process p ψ ( x t − 1 | x t ) , we parametrize the timestep-dependent noise function ϵ ψ ( x t , t ) with a neural network and learn it by optimizing a simplified version of the variational lower bound on p ( x )
[Equation]
Sampling from the reverse process can then be done with
[Equation]
with ξ ∼ N (0 , I ) and σ t chosen between β t and β t / √ 1 − ¯ α t .
Sampling from the learned diffusion model can be seen as analogue to Langevin Dynamics, a connection explicitly made in works exploring the diffusion technique from the “score matching” perspective (Song and Ermon, 2019;
Song et al., 2020).
A.6 Classifier-Free Guidance
We hereby provide a rationale for the use of classifier guidance and classifier-free guidance during diffusion model sampling.
As per the “score matching” interpretation of diffusion models, we assume that our trained noise network approximates the score function of the true conditional latent distribution p ( z | e i ) as ϵ ψ ( z t , t, e i )
≈ − σ t ∇ z t log p ( z t | e i ) .
For classifier guidance, we can perturb our diffusion sampling by adding the gradient of the log likelihood of our CLIP encoder p ψ
( e i | z t ) to the diffusion score as follows
[Equation]
We can rewrite this as classifier guidance on the unconditional score ∇ z t log p ( z t ) with
[Equation]
with
[Equation]
[Equation]
using Bayes’ rule, as log p ( z t | e i )
= log p ( e i | z t )+log p ( z t ) − log p ( e i ) , and thus ∇ z t log p ( z t | e i ) =
∇ z t log p ( e i | z t ) + ∇ z t log p ( z t ) .
For classifier-free guidance, we aim to perform the above sampling without access to a classifier, as long we possess a conditional diffusion model ϵ ψ ( z t , t, e i ) that doubles as an unconditional model ϵ ψ ( z t , t, 0) , as illustrated in Section 4.
Using Bayes’ rule again, we can see that ∇ z t log p ( e i | z t ) = ∇ z t log p ( z t | e i )
− ∇ z t log p ( z t ) .
If we substitute this into Eq.
11 we obtain
[Equation]
which can be implemented with our conditional network as
[Equation]
A.7 Network architectures
Base Network ( f ) Our choice for a base model is a CLIP-Adapter (Gao et al., 2021), which consists of a frozen CLIP image encoder with added learned fully-connected layers refining the output embedding.
Specifically, we use the ViT-L/14@336px CLIP encoder type with embedding size of 768.
The advantages of this model choice lie in its combination of high base performance (due to pre-trained knowledge contained in the CLIP component) and relatively small parameter count, enabling agile medium-small scale experiments.
This base CLIP-Adapter network purely works as a base model and is not to be confused with HyperCLIP, which is employed at the meta-level.
In
Section 5, when benchmarking the base model in the zero-shot setting, we drop the Adapter and use pre-trained zero-shot CLIP (Radford et al., 2021).
Hypernetwork ( h ) For the hypernetworks used in our baseline as well as as the generative model, we use a MLP with one hidden layer of 256 units, which are followed by a rectified linear activation.
For the unconditioned hypernetwork, the embedding to the hypernetwork is a vector of dimension 64, while for the conditioned counterpart, the task embedding is used.
In order to ensure that the generated weights are properly normalized at initialization, we use the Kaiming initialization (He et al., 2015) for the hypernetwork weights, initialize the embedding as a sample from a multivariate standard gaussian distribution (for unconditioned models), and use the NTK parametrization (Jacot et al., 2020) for the target network.
Variational Autoencoder For the variational autoencoder used as our unconditioned generative model, we use an MLP of 2 hidden layers of size 512 and 256, each followed by the rectified linear non-linearity.
We chose 32 as the latent code dimension.
We use the same architecture for the decoder, except for the dimensionality of the 2 hidden layers being swapped.
We use the Kaiming initialization (He et al., 2015) to initialize the weight of both the encoder and decoder.
HyperCLIP We parametrize our HyperCLIP model as a fully-connected MLP with a single hidden layer of dimension 256, taking as input the flattened weight of the base network and outputting the corresponding CLIP encoding.
We chose the tangent hyperbolic function as the activation function in the hidden layer.
HyperLDM While the original LDM makes use of a time-conditional UNet (Ronneberger et al., 2015) to parametrize the noise network, we are unfortunately unable to make use of spatial information and convolutions due to the non-spatial nature of our latent space.
We parametrize our HyperLDM as a fully-connected network with residual connections and squeeze-and-excitation layers (Hu et al., 2018).
The time index t is embedded into a vector with a 150-dimensional sinusoidal positional embedding, and is concatenated together with the task-conditional embedding e i at the input layer and at intermediate activations.
Hidden layer dimensions are 8192, 16384, 8192.
A.8 The Meta-VQA Dataset
To evaluate the performance of our methods, we utilize a dataset that reflects the setting of metalearning with multi-modal task descriptors.
Existing meta-learning benchmarks such as MiniImagenet (Ravi and Larochelle, 2016) or CIFAR-FS (Bertinetto et al., 2018) are unsuitable, as they are built for the traditional few-shot learning setting, in which the task T i is not associated with task descriptors but is meant to be inferred through exposure to the support set D s i .
We thus introduce our own Meta-VQA dataset, a modification of the VQA v2.0 dataset (Goyal et al., 2017) for Visual-Question-Answering.
The dataset is composed of training and test tasks T i , each associated with a natural language question t i and a mini image classification dataset ( x ij , y ij ) ∈
D i .
The original VQA problem is about choosing a suitable natural language answer a k when prompted with both a natural language question q k and an image
I k .
Our observation is that the VQA problem can then easily be reformulated as a meta-learning image classification problem with natural language task descriptions: given question-image-answer triples ( q k , I k , a k ) ∈ D , we can group the data by unique questions q i (which will serve as task descriptor t i ), each of which can then be associated with supervised image classification tuples ( I ij , a ij ) ∈ D i .
To make sure the designed tasks are meaningful, we filter out question-answer pairs with questions in choosing form, e.g., “A or B?” or yes/no answers.
From the remaining questions we keep the ones which appear at least 20 times throughout the dataset, such that each task contains enough samples.
In the end our Meta-VQA dataset is composed of 1234 unique tasks (questions), split into 870 training tasks and 373 test tasks, for a total ofimage-answer pairs.
A.9 Experimental Results
We point to Table 1 for a recount of zero-shot and few-shot classification accuracy obtained on
Meta-VQA test tasks with both the baselines and our techniques.
For each training task T i , the algorithms are given access to the full image/answer support and query sets D s i , D q i , together with the question (task descriptor)
t i .
At test time, in the zero-shot setting, only the task descriptors t i for each test task T i are given, and the algorithms are tasked with predicting the correct labels of images in the query set D q i .
In the few-shot setting, beyond the task descriptors t i , the support sets D s i of test tasks with accompanying answers (labels) are also given.
[Table]
A.10 Baseline methods
Classic zero-shot CLIP and its CLIP-Adapter extension for few-shot learning (our base network) provide a floor for performance on Meta-VQA.
We then benchmark several unconditional and conditional methods, with only conditional methods having access to language task descriptors.
We apply MAML and its first-order variant FOMAML (Nichol et al., 2018) directly to the base network ( MNet-MAML , MNet-FOMAML ), and to both an unconditional hypernetwork ( Uncond.
HNetMAML , as in Appendix A.3) and a conditional one ( Cond.
HNet-MAML ).
We also benchmark against standard multitask learning ( Uncond.
Multitask , Cond.
Multitask ).
It is apparent that the multitask approach, at least in this setting, leads to better zero-shot models than MAML, which instead optimizes for few-shot performance.
We refer to Appendix A.7 for more details on each model.
We detail an overview of the baseline methods we benchmark in table 2.
Training:
The number of epochs each model is trained on, the learning rate lr of the optimization, as well as the learning rate and number of steps of the adaptation algorithm used for each method can be found in table 3.
For all methods using an adaptation A T i , the dataset from the task is randomly split into a support set and a query set during training, every time a task is sampled.
The support set is then used to perform the adaptation (see Section A.2), while the query set is used to compute the loss on which the meta-parameters are updated.
When no adaptation is used, all the data is used for this update.
Unconditional methods do not have access to the task embedding e i , while conditioned methods do.
When the percentage of available task descriptor is reduced, conditioned methods are trained only on the tasks which descriptor is available.
Evaluation: Evaluation is performed on a fixed query set on the predefined query set of the heldout test tasks of the Meta-VQA dataset.
Zero-shot performance is evaluated before applying the adaptation procedure A T i .
For the few shot performance, all adaptation is performed on the support set of the test tasks.
For MAML baselines, we keep the same adaptaiton-time learning rate as during trainng, while we always adapt for 50 steps.
For each multitask baselines, we use the same adaptation scheme (steps, learning rate, adapting parameters) as their MAML counterpart.
A.11 Guidance Models Hyperparameters A.11.1 Generative hypernetwork
To enable our guidance methods, we need to first train a generative hypernetwork h as in Section ??
, either in the form of an Unconditional Hypernetwork, or of a Hypernetwork VAE:
[Table]
[Table]
• For HNet + HyperCLIP guidance
and HNet + HyperLDM , we meta-learnt an unconditioned hypernetwork with the exact same hyperparameters as the baseline Uncond.
HNetMAML , and used it as the generative hypernetwork.
• For HVAE + HyperCLIP guidance and HVAE + HyperLDM , we trained an unconditioned VAE on samples of fine tuned network weights W i using the architecture specified in A.7.
In order to be able to quickly sample new adapted weights, and to reduce the complexity of the manifold from which such weights are sampled, we use adaptations from our unconditional MAML baselines as W i .
Specifically, we used adaptations from Uncond.
HNet-MAML , using 50-step adaptation A T i with learning rate 0 .
1 , on support set stochastically sampled for every adaptation phase.
We trained the VAE on 2000 epochs where each epoch is a single pass through all the tasks, with the Adam (Kingma and Ba, 2017) optimizer and 0 .
0001 learning rate and batch size 32.
We used gradient norm clipping independently for both the encoder and decoder, with the maximum norm capped at 1000.
A.11.2 HyperCLIP
Training In order to train the HyperCLIP model, we need samples of fine tuned network weights
W i .
Similarly to HVAE, we used adaptations from Uncond.
HNet-MAML , using 50-step adaptation
A T i with learning rate 0 .
1 , on a support set stochastically sampled at every adaptation phase, as this would allow us to use the same HyperCLIP model for doing guidance on both HNet and HVAE.
We trained our HyperCLIP model for 600 epochs, with the Adam (Kingma and Ba, 2017) optimizer, 0 .
0003 learning rate and batch size 64, for all our experiments.
Guidance We use 10 steps guidance with λ = 0 .
01 and learning rate 0 .
1 , for both when performed on HNet and HVAE.
Evaluation Evaluation is performed on a fixed query set on the predefined query set of the heldout test tasks of the Meta-VQA dataset.
Zero-shot performance is evaluated on the output of the generative hypernetwork h after applying latent space guidance.
For the few shot performance, all adaptation is performed on the support set of the test tasks, on the latent space initialized at the output of the guidance procedure.
Similarly to our baselines, we use 50-steps gradient descent adaptation with learning rate 0.1.
A.11.3 HyperLDM
Training Similarly to HyperCLIP, to train HyperLDM we need samples of fine tuned network weights W i , for which we use adaptations from Uncond.
HNet-MAML , using 50-step adaptation
A T i with learning rate 0 .
1 , on a support set stochastically sampled at every adaptation phase.
We parametrize the diffusion process with a linear noise schedule, β starting at 0.0001 and ending at 0.06, and 350 diffusion timesteps.
We train the HyperLDM for 1000 epochs with the Adam optimizer, 0 .
00025 learning rate and 128 epochs, for all our experiments.
Evaluation Evaluation is performed as for HyperCLIP guidance, except for the fact that adaptation is performed natively through sampling from the learned reversed diffusion process, with parameters derived from the chosen β schedule.
The guidance parameter γ > 0 can be tuned during inference to accentuate the effect of classifier-free guidance.