Original: H YPER T RANSFORMER : A TTENTION -B ASED CNN M ODEL G ENERATION FROM F EW S AMPLES
Revised: H YPER T RANSFORMER : A TTENTION -B ASED CNN M ODEL G ENERATION FROM F EW S AMPLES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this work we propose a HyperTransformer  a transformer based model that generates all weights of a CNN model directly fromthe support samples
Revised: In this work we propose a HyperTransformer , a transformer based model that generates all weights of a CNN network directly from support samples.

Original: This ap- proach allows to use a high-capacity model for encoding task-dependent variations in the weights of a smaller model.
Revised: This approach uses a high-capacity model for encoding task-dependent variations in the weights of a smaller model.

Original: We show for multiple few-shot  benchmarks with different architectures and datasets that our method beats or matchesthat of the  traditional learning methods in a few-shot regime.
Revised: We show on multiple few-shot image classiﬁcation benchmarks with different model sizes and datasets that our method beats or matches the performance of many traditional learning methods.

Original: Speciﬁcally, we show that for very small target models, our method can generate signiﬁcantly better performing models than traditional few-shot learning methods.
Revised: Speciﬁcally, we show that for very small target architectures, our method can generate signiﬁcantly better performing models than traditional few-shot learning approaches.

Original: For larger models we discover that applying generation to the last layer only, allows  to produce competitive or better results while being end-to-end differentiable.
Revised: For larger models we discover that generating the last layer alone allows us to produce competitive or better results while being end-to-end differentiable.

Original: Finally, we extend our approach to  semi-supervised regime utilizing unlabeled samples in the support set and further improving few-shot performance in the presence of unlabeled data
Revised: Finally, we extend our approach to a semi-supervised regime utilizing unlabeled samples in the support set and further improving few-shot performance in the presence of unlabeled data

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In few-shot learning, a conventional machine learning paradigm of ﬁtting a parametric model to training data is taken to a limit of extreme data scarcity where entire categories are introduced with just one or few examples.
Revised: In few-shot learning, a conventional machine learning paradigm of ﬁtting a parametric model to training data is taken to a limit of extreme data scarcity where entire categories are introduced with just one or few examples.

Original: A generic approach to solving this problem uses training data to identify parameters φ of a learner a φ that given a small batch of examples for a particular task (called a support set) can solve this task on unseen data (called a query set).
Revised: A generic approach to solving this problem uses training data to identify parameters φ of a learner a φ that given a small batch of examples for a particular task (called a support set) can solve this task on unseen data (called a query set).

Original: One broad family of few-shot  classiﬁcation methods frequently referred to as metric-based learning , relies on pretraining a low-dimensional embedding e φ ( · ) and then using some distance in the embedding space to label query samples based on their closeness to known labeled support samples.
Revised: One broad family of few-shot image classiﬁcation methods frequently referred to as metric-based learning , relies on pretraining an embedding e φ ( · ) and then using some distance in the embedding space to label query samples based on their closeness to known labeled support samples.

Original: These methods proved effective on numerous benchmarks (see Tian et al.
Revised: These methods proved effective on numerous benchmarks (see Tian et al.

Original: (2020) for review and references), however the capabilities of the learner are limited by the capacity of the architecture itself, as these methods try to build a universal embedding function.
Revised: (2020) for review and references), however the capabilities of the learner are limited by the capacity of the architecture itself, as these methods try to build a universal embedding function.

Original: On the other hand, optimization-based methods such as seminal MAML algorithm
Revised: On the other hand, optimization-based methods such as seminal MAML algorithm

Original: (Finn et al., 2017) can ﬁne-tune the embedding e φ by performing additional SGD updates on all parameters φ of the model producing it.
Revised: (Finn et al., 2017) can ﬁne-tune the embedding e φ by performing additional SGD updates on all parameters φ of the model producing it.

Original: This partially addresses the constraints of metric-based methods by learning a new embedding for each new task.
Revised: This partially addresses the constraints of metric-based methods by learning a new embedding for each new task.

Original: However, in many of these methods, allof the knowledge extracted during training on different tasks and describing the learner a φ still has to “ﬁt” into the same number of parameters as the model itself.
Revised: However, in many of these methods, all the knowledge extracted during training on different tasks and describing the learner a φ still has to “ﬁt” into the same number of parameters as the model itself.

Original: Such limitation becomes more severe as the target models get smaller, while the richness of the task set increases.
Revised: Such limitation becomes more severe as the target models get smaller, while the richness of the task set increases.

Original: In this paper we propose a new few-shot learning approach that allows  to decouple the complexity of the task space from the complexity of individual tasks.
Revised: In this paper we propose a new few-shot learning approach that allows us to decouple the complexity of the task space from the complexity of individual tasks.

Original: The main idea is to use  transformer model (Vaswani et al., 2017) that given  few-shot task episode generates an entire inference model by producing all model weights in a single pass.
Revised: The main idea is to use the transformer model (Vaswani et al., 2017) that given a few-shot task episode generates an entire inference model by producing all model weights in a single pass.

Original: This allows  to encode the intricacies of the available training data inside the transformer model, while still producing specialized tiny models that can solve individual tasks.
Revised: This allows us to encode the intricacies of the available training data inside the transformer model, while still producing specialized tiny models that can solve individual tasks.

Revised: Reducing the size of the generated model and moving the computational overhead to the transformer-based weight generator, we can lower the cost of the inference on new images.

Revised: This can dramatically reduce the overall computation cost in cases, where the tasks change infrequently and hence the weight generator is only used sporadically.

Original: We start by observing that the self-attention mechanism is well suited to be the underlying mechanism for the few-shot CNN weight generator.
Revised: We start by observing that the self-attention mechanism is well suited to be an underlying mechanism for a few-shot CNN weight generator.

Original: In contrast with earlier CNN- (Zhao et al., 2020) or BiLSTM-based approaches (Ravi & Larochelle, 2017), the  transformer model is invariant to sample permutations and can handle unbalanced datasets with a varying number of samples per category.
Revised: In contrast with earlier CNN- (Zhao et al., 2020) or BiLSTM-based approaches (Ravi & Larochelle, 2017), the vanilla 1 transformer model is invariant to sample permutations and can handle unbalanced datasets with a varying number of samples per category.

Original: Furthermore, we demonstrate that a single-layer self-attention model can replicate a sim- pliﬁed gradient-descent-based learning algorithm.
Revised: Furthermore, we demonstrate that a single-layer self-attention model can replicate a simpliﬁed gradient-descent-based learning algorithm.

Revised: Using a transformer-based model to generate the logits layer on top of a conventionally learned embedding, we achieve competitive results on several common few-shot learning benchmarks.

Original: Varying transformer parameters we demonstrate that this high performance can be attributed to additional capacity of the transformer model that decouples its complexity from that of the generated CNN.
Revised: Varying transformer parameters we demonstrate that this high performance can be attributed to additional capacity of the transformer model that decouples its complexity from that of the generated CNN.

Original: We then extend our method to support unlabeled samples by using  special input token concatenated to unlabeled samples to encode unknown labels.
Revised: We then extend our method to support unlabeled samples by using a special input token concatenated to unlabeled samples to encode unknown labels.

Original: In our experiments, we observe that using transformers with two or more layers, we achieve better performance by adding unlabeled data into the support set.
Revised: In our experiments, we observe that using transformers with two or more layers, we achieve better performance by adding unlabeled data into the support set.

Original: We explain our results in Section 4.3 where we show that such  transformer with at least two layers can encode  nearest-neighbor style algorithm that associates unlabeled samples with similar labeled examples.
Revised: We explain our results in Section 4.3 where we show that such a transformer with at least two layers can encode the nearest-neighbor style algorithm that associates unlabeled samples with similarly labeled examples.

Original: In essence, by training the weight generator to produce CNN models with best possible performance on a query set, we teach the transformer to utilize unlabeled samples without having to manually introduce additional optimization objectives.
Revised: In essence, by training the weight generator to produce CNN models with best possible performance on a query set, we teach the transformer to utilize unlabeled samples without having to manually introduce additional optimization objectives.

Original: We  show that by generating all layers of the CNN model we can improve both the training and  test accuracies of CNN models below a certain size.
Revised: We also show that by generating all layers of the CNN model we can improve both the training and the test accuracies of CNN models below a certain size.

Original: The training accuracy can be viewed as a capability of the generated CNN model to adapt to tasks seen at training time, whereas the test accuracy computed on unseen categories characterizes the generalization capability of this model adaptation mechanism.
Revised: The training accuracy can be viewed as a capability of the generated CNN model to adapt to tasks seen at training time, whereas the test accuracy computed on unseen categories characterizes the generalization capability of this model adaptation mechanism.

Original: We empirically demonstrate the expected increase of the model training and test accuracies with the increase of the layer size and the number of generated layers (see Figure 3).
Revised: We empirically demonstrate the expected increase of the model training and test accuracies with the increase of the layer size and the number of generated layers (see Figure 3).

Original: Interestingly, generation of the logits layer alone appears to be “sufﬁcient” above a certain model size threshold.
Revised: Interestingly, generation of the logits layer alone appears to be “sufﬁcient” above a certain model size threshold.

Original: This threshold is expected to depend on the variability and the complexity of the training tasks.
Revised: This threshold is expected to depend on the variability and the complexity of the training tasks.

Original: We conjecture that this might reﬂect the fact that models are sufﬁciently expressive for the benchmarks we considered, or that additional regularization is needed to prevent overﬁtting of the meta-learning models.
Revised: We conjecture that this might reﬂect the fact that models are sufﬁciently expressive for the benchmarks we considered, or that additional regularization is needed to prevent overﬁtting of the meta-learning models.

Original: Finally, in addition to being able to decouple the complexity of the task distribution from the complexity of individual tasks, another important advantage of our method is that it allows to do learning end-to-end without relying on complex nested gradients optimization and other meta-learning approaches, where the number of unrolls steps is large.
Revised: Finally, in addition to being able to decouple the complexity of the task distribution from the complexity of individual tasks, another important advantage of our method is that it allows to do learning end-to-end without relying on complex nested gradients optimization and other meta-learning approaches, where the number of unrolls steps is large.

Original: The paper is structured as follows.
Revised: The paper is structured as follows.

Original: In Section 2, we discuss the few-shot learning problem setup and highlight related work.
Revised: In Section 2, we discuss the few-shot learning problem setup and highlight related work.

Original: Section 3 introduces our approach, discusses the motivation for choosing an attention-based model and shows how our approach can be used to meta-learn semi-supervised learning algorithms.
Revised: Section 3 introduces our approach, discusses the motivation for choosing an attention-based model and shows how our approach can be used to meta-learn semi-supervised learning algorithms.

Original: In Section 4, we discuss our experimental results.
Revised: In Section 4, we discuss our experimental results.

Original: Finally, in Section 5, we provide concluding remarks.
Revised: Finally, in Section 5, we provide concluding remarks.

Original:  P ROBLEM S ETUP AND R ELATED W ORK 2. F EW -S HOT L EARNING
Revised:  P ROBLEM S ETUP AND R ELATED W ORK 2. F EW -S HOT L EARNING

Original: The main goal of a few-shot learning algorithm is to use a set of training tasks T train for ﬁnding a learner a φ parameterized by φ that given new task domains can train to recognize novel classes using just a few samples per each class.
Revised: The main goal of a few-shot learning algorithm is to use a set of training tasks T train for ﬁnding a learner a φ parameterized by φ that given new task domains can train to recognize novel classes using just a few samples per each class.

Original: The learner a φ can be thought of as a function that maps task description T = { ( x i , c i ) } ti =1 containing k labeled input samples { x i , c i } from n classes, to the weights θ = a φ ( T ) of a trained model f
Revised: The learner a φ can be thought of as a function that maps task description T = { ( x i , c i ) } ti =1 containing k labeled input samples { x i , c i } from n classes, to the weights θ = a φ ( T ) of a trained model f

Original: ( x ; θ ) .
Revised: ( x ; θ ) .

Original: The parameters φ are meta-optimized to maximize the performance of the model f
Revised: The parameters φ are meta-optimized to maximize the performance of the model f

Original: ( x ; a φ ( T S )) generated using a support set T S with x drawn from a query set T Q .
Revised: ( x ; a φ ( T S )) generated using a support set T S with x drawn from a query set T Q .

Original: Each task T =
Revised: Each task T =

Original: ( T S , T Q ) is randomly drawn from a space of training tasks T train .
Revised: ( T S , T Q ) is randomly drawn from a space of training tasks T train .

Original: Typically, T S and T Q are generated by ﬁrst randomly choosing several distinct classes from the training set and then sampling examples without replacement from these classes to generate T S and
Revised: Typically, T S and T Q are generated by ﬁrst randomly choosing several distinct classes from the training set and then sampling examples without replacement from these classes to generate T S and

Original: T Q .
Revised: T Q .

Original: In a classical “ n -way- k -shot” setting, n is the number of classes randomly sampled in each episode, and k is the number of samples for each class in the support set T S .
Revised: In a classical “ n -way- k -shot” setting, n is the number of classes randomly sampled in each episode, and k is the number of samples for each class in the support set T S .

Original: The quality of a particular few-shot learning algorithm is typically evaluated using a separate test space of tasks T test .
Revised: The quality of a particular few-shot learning algorithm is typically evaluated using a separate test space of tasks T test .

Original: By forming T test from novel classes unseen at training time, we can evaluate generalization of different learners a φ .
Revised: By forming T test from novel classes unseen at training time, we can evaluate generalization of different learners a φ .

Original: Best algorithms are expected to capture the structure present in the training set and to perform well on novel concepts.
Revised: Best algorithms are expected to capture the structure present in the training set and to perform well on novel concepts.

Original: This structure may, for example, include certain properties of the distributions p c ( x ) with c being the class label, or the presence of particular discriminative (or alternatively invariant) features in the tasks from T train .
Revised: This structure may, for example, include certain properties of the distributions p c ( x ) with c being the class label, or the presence of particular discriminative (or alternatively invariant) features in the tasks from T train .

Original: 2. R ELATED WORK
Revised: 2. R ELATED WORK

Original: Few-shot learning received a lot of attention from the deep learning community and while there are hundreds of few-shot learning methods, several common themes emerged in the past years.
Revised: Few-shot learning received a lot of attention from the deep learning community and while there are hundreds of few-shot learning methods, several common themes emerged in the past years.

Original: Here we outline several existing approaches, show how they relate to our method and discuss the prior work related to it.
Revised: Here we outline several existing approaches, show how they relate to our method and discuss the prior work related to it.

Original: Metric-Based Learning.
Revised: Metric-Based Learning.

Original: One family of approaches involves mapping input samples into a lowerdimensional embedding space and then using some nearest neighbor algorithm that relies on the computation of distances from a query sample embedding to the embedding computed using support samples with known labels.
Revised: One family of approaches involves mapping input samples into an embedding space and then using some nearest neighbor algorithm that relies on the computation of distances from a query sample embedding to the embedding computed using support samples with known labels.

Original: The metric used to compute the distance can either be the same for all tasks, or can be task-dependent.
Revised: The metric used to compute the distance can either be the same for all tasks, or can be task-dependent.

Original: This family of methods includes, for example, such methods as Siamese networks (Koch et al., 2015), Matching Networks (Vinyals et al., 2016), Prototypical Networks (Snell et al., 2017), Relation Networks (Sung et al., 2018) and TADAM (Oreshkin et al., 2018).
Revised: This family of methods includes, for example, such methods as Siamese networks (Koch et al., 2015), Matching Networks (Vinyals et al., 2016), Prototypical Networks (Snell et al., 2017), Relation Networks (Sung et al., 2018) and TADAM (Oreshkin et al., 2018).

Original: It has recently been argued (Tian et al., 2020) that methods based on building a powerful sample representation can frequently outperform numerous other approaches including many optimization-based methods.
Revised: It has recently been argued (Tian et al., 2020) that methods based on building a powerful sample representation can frequently outperform numerous other approaches including many optimization-based methods.

Original: However, such approaches essentially amount to  “one-model solves all” approach and thus require larger models than needed to solve individual tasks.
Revised: However, such approaches essentially amount to the “one-model solves all” approach and thus require larger models than needed to solve individual tasks.

Original: Optimization-Based Learning.
Revised: Optimization-Based Learning.

Original: An alternative approach that can adapt the embedding to a new task is to incorporate optimization within the learning process.
Revised: An alternative approach that can adapt the embedding to a new task is to incorporate optimization within the learning process.

Original: A variety of such methods are based on the approach called Model-Agnostic Meta-Learning , or MAML (Finn et al., 2017).
Revised: A variety of such methods are based on the approach called Model-Agnostic Meta-Learning , or MAML (Finn et al., 2017).

Original: In MAML, θ = a φ is obtained by initializing a DNN at θ 0 = φ
Revised: In MAML, θ = a φ is obtained by initializing a DNN at θ 0 = φ

Original: and then performing one or more gradient descent updates on a classiﬁcation loss function L , i.e., computing θ k +

Revised: and then performing one or more gradient descent updates on a classiﬁcation loss function L , i.e., computing θ k +

Original: =
Revised: =

Original: θ k − γ · ( ∂L/∂ θ )( T ; θ k ) .
Revised: θ k − γ · ( ∂L/∂ θ )( T ; θ k ) .

Original: This approach was later reﬁned (Antoniou et al., 2019) and built upon giving rise to Reptile (Nichol et al., 2018), LEO (Rusu et al., 2019) and others.
Revised: This approach was later reﬁned (Antoniou et al., 2019) and built upon giving rise to Reptile (Nichol et al., 2018), LEO (Rusu et al., 2019) and others.

Original: One limitation of various MAML-inspired methods is that the knowledge about the set of training tasks T train is distilled into parameters φ that have the same dimensionality as the model parameters θ .
Revised: One limitation of various MAML-inspired methods is that the knowledge about the set of training tasks T train is distilled into parameters φ that have the same dimensionality as the model parameters θ .

Original: Therefore, for  very lightweight model f ( x ; θ ) the capacity of the task-adaptation learner a φ is still limited by the size of θ .
Revised: Therefore, for a very lightweight model f ( x ; θ ) the capacity of the task-adaptation learner a φ is still limited by the size of θ .

Original: Methods that use parameterized preconditioners that otherwise do not impact the model f ( x ; θ ) can alleviate this issue, but as with MAML, such methods can be difﬁcult to train (Antoniou et al., 2019).
Revised: Methods that use parameterized preconditioners that otherwise do not impact the model f ( x ; θ ) can alleviate this issue, but as with MAML, such methods can be difﬁcult to train (Antoniou et al., 2019).

Original: Weight Modulation and Generation.
Revised: Weight Modulation and Generation.

Original: The idea of using a task speciﬁcation to directly generate or modulate model weights has been previously explored in the generalized supervised learning context (Ratzlaff & Li, 2019) and in speciﬁc language models (Mahabadi et al., 2021; Tay et al., 2021; Ye & Ren, 2021).
Revised: The idea of using a task speciﬁcation to directly generate or modulate model weights has been previously explored in the generalized supervised learning context (Ratzlaff & Li, 2019) and in speciﬁc language models (Mahabadi et al., 2021; Tay et al., 2021; Ye & Ren, 2021).

Original: Someof the few-shot learning methods described above also employ this approach and use task-speciﬁc generation or modulation of the weights of the ﬁnal classiﬁcation model.
Revised: Some few-shot learning methods described above also employ this approach and use task-speciﬁc generation or modulation of the weights of the ﬁnal classiﬁcation model.

Original: For example, in LGM-Net (Li et al., 2019b)
Revised: For example, in LGM-Net (Li et al., 2019b)

Original: the matching network approach is used to generate a few layers on top of a task-agnostic embedding.
Revised: the matching network approach is used to generate a few layers on top of a task-agnostic embedding.

Original: Another approach abbreviated as LEO (Rusu et al., 2019) utilized  similar weight generation method to generate initial model weights from the training dataset in a few-shot learning setting, much like what is proposed in this article.
Revised: Another approach abbreviated as LEO (Rusu et al., 2019) utilized a similar weight generation method to generate initial model weights from the training dataset in a few-shot learning setting, much like what is proposed in this article.

Original: However, in (Rusu et al., 2019), the generated weights were also reﬁned using several SGD steps similar to how it is done in MAML.
Revised: However, in (Rusu et al., 2019), the generated weights were also reﬁned using several SGD steps similar to how it is done in MAML.

Original: Here we explore a similar idea, but largely inspired by the H YPER N ETWORK approach (Ha et al., 2017), we instead propose to directly generate an entire task-speciﬁc CNN model.
Revised: Here we explore a similar idea, but largely inspired by the H YPER N ETWORK approach (Ha et al., 2017), we instead propose to directly generate an entire task-speciﬁc CNN model.

Original: Unlike LEO, we do not rely on pre-computed embeddings for images and generate the model in a single step without additional SGD steps, which simpliﬁes and stabilizes training.
Revised: Unlike LEO, we do not rely on pre-computed embeddings for images and generate the model in a single step without additional SGD steps, which simpliﬁes and stabilizes training.

Original: Transformers in Computer Vision and Few-Shot Learning.
Revised: Transformers in Computer Vision and Few-Shot Learning.

Original: Transformer models (Vaswani et al., 2017) originally proposed for natural language understanding applications had since became a useful tool in practically every subﬁeld of deep learning.
Revised: Transformer models (Vaswani et al., 2017) originally proposed for natural language understanding applications had since become a useful tool in practically every subﬁeld of deep learning.

Original: In computer vision, transformers have recently seen an explosion of applications ranging from state-of-the-art image classiﬁcation results (Dosovitskiy et al., 2021; Touvron et al., 2021) to object detection (Carion et al., 2020; Zhu et al., 2021), segmentation (Ye et al., 2019), image super-resolution (Yang et al., 2020), image generation  (Chen et al., 2021) and many others.
Revised: In computer vision, transformers have recently seen an explosion of applications ranging from state-of-the-art image classiﬁcation results (Dosovitskiy et al., 2021; Touvron et al., 2021) to object detection (Carion et al., 2020; Zhu et al., 2021), segmentation (Ye et al., 2019), image super-resolution (Yang et al., 2020), image generation [Figure] (Chen et al., 2021) and many others.

Original: There are also several notable applications in few-shot image classiﬁcation.
Revised: There are also several notable applications in few-shot image classiﬁcation.

Original: For example, in Liu et al.
Revised: For example, in Liu et al.

Original: (2021), the transformer model was used for generating universal representations in the multi-domain few-shot learning scenario.
Revised: (2021), the transformer model was used for generating universal representations in the multi-domain few-shot learning scenario.

Original: And closely related to our approach, in Ye et al.
Revised: And closely related to our approach, in Ye et al.

Original: (2020), the authors proposed to accomplish embedding adaptation with the help of transformer models.
Revised: (2020), the authors proposed to accomplish embedding adaptation with the help of transformer models.

Original: Unlike our method that generates an entire end-to-end image classiﬁcation model, this approach applied a task-dependent perturbation to an embedding generated[Figure] by an independent task-agnostic feature extractor.
Revised: Unlike our method that generates an entire end-to-end image classiﬁcation model, this approach applied a task-dependent perturbation to an embedding generated by an independent task-agnostic feature extractor.

Original: In (Gidaris & Komodakis, 2018), a simpliﬁed attention-based model was used for the ﬁnal layer generation.
Revised: In (Gidaris & Komodakis, 2018), a simpliﬁed attention-based model was used for the ﬁnal layer generation.

Original:  O UR A PPROACH
Revised:  O UR A PPROACH

Original: In this section we describea our approach to few-shot learning that we call a H YPER T RANSFORMER (HT) and justify the choice of  self-attention mechanism as its basis.
Revised: In this section, we describe our approach to few-shot learning that we call a H YPER T RANSFORMER (HT) and justify the choice of the self-attention mechanism as its basis.

Original: 3. F EW -S HOT L EARNING M ODEL
Revised: 3. F EW -S HOT L EARNING M ODEL

Original: A learner a φ (as introduced in Section 2.1) is the core of a few-shot learning algorithm and in this paper, we choose a φ to be a transformer-based model that takes a task description T
Revised: A learner a φ (as introduced in Section 2.1) is the core of a few-shot learning algorithm and in this paper, we choose a φ to be a transformer-based model that takes a task description T

Original: = { ( x i , c i ) } ti = as input and produces weights for some or all layers { θ (cid:96) | (cid:96) = 1 , . . . , L  } of the generated CNN model.
Revised: = { ( x i , c i ) } ti = as input and produces weights for some or all layers { θ (cid:96) | (cid:96) ∈ [1 , L ] } of the generated CNN model.

Original: For layers with non-generated weights, they are learned in the end-to-end fashion as ordinary taskagnostic variables.
Revised: For layers with non-generated weights, they are learned in the end-to-end fashion as ordinary taskagnostic variables.

Original: In our experiments generated CNN models contain a set of convolutional layers and a ﬁnal fully-connected logits layer.
Revised: In our experiments generated CNN models contain a set of convolutional layers and a ﬁnal fully-connected logits layer.

Original: Here θ (cid:96) are the parameters of the (cid:96) -th layer and L is the total number of layers including the ﬁnal logits layer (with index L ).
Revised: Here θ (cid:96) are the parameters of the (cid:96) -th layer and L is the total number of layers including the ﬁnal logits layer (with index L ).

Original: The weights are generated layer by layer starting from the ﬁrst layer: θ 1 ( T ) → θ 2 ( θ 1 ; T ) → · · · → θ L ( θ 1 ,...,L − 1 ; T ) .
Revised: The weights are generated layer by layer starting from the ﬁrst layer: θ 1 ( T ) → θ 2 ( θ 1 ; T ) → · · · → θ L ( θ 1 ,...,L − 1 ; T ) .

Original: Here we use θ a,...,b as a short notation for ( θ a , θ a +1 , . . .
Revised: Here we use θ a,...,b as a short notation for ( θ a , θ a +1 , . . .

Original: , θ b ) .
Revised: , θ b ) .

Original: Shared and local features.
Revised: Shared and local features.

Original: The local features at layer (cid:96) are produced by a convolutional feature extractor h (cid:96) φ l ( z (cid:96)i ) applied to the activations of the previous layer z (cid:96)i :=
Revised: The local features at layer (cid:96) are produced by a convolutional feature extractor h (cid:96) φ l ( z (cid:96) i ) applied to the activations of the previous layer z (cid:96) i :=

Original: f (cid:96) − 1 ( x i ; θ 1 ,...,(cid:96) − 1 ) for (cid:96) >and z 1 i := x i .
Revised: f (cid:96) − 1 ( x i ; θ 1 ,...,(cid:96) − 1 ) for (cid:96) >and z 1 i := x i .

Original: In other words, the transformer for layer (cid:96) receives
Revised: In other words, the transformer for layer (cid:96) receives

Original: [Equation]
Revised: [Equation]

Original: The intuition behind the local feature extractor is that the choice of the layer weights should primarily depend on the inputs received by this layer.
Revised: The intuition behind the local feature extractor is that the choice of the layer weights should primarily depend on the inputs received by this layer.

Original: The shared features, on the other hand, are the same  for all layers and are produced by a separate trainable convolutional neural network s φ s ( x i ) .
Revised: The shared features, on the other hand, are the same [Figure] for all layers and are produced by a separate trainable convolutional neural network s φ s ( x i ) .

Original: Their purpose is to modulate each layer’s weight generator with a global high-level sample embedding that, unlike the local embedding, is independent of the generated weights and is also fully shared between all layer generators.[Figure] (i) (ii)
Revised: Their purpose is to modulate each layer’s weight generator with a global high-level sample embedding that, unlike the local embedding, is independent of the generated weights and is also fully shared between all layer generators.

Original: Encoding and decoding transformer inputs and outputs.
Revised: Encoding and decoding transformer inputs and outputs.

Original: In our experiments we considered several different architectures of the transformer model, different ways of feeding I (cid:96) into it and different ways of converting transformer outputs into the CNN weights.
Revised: In our experiments we considered several different architectures of the transformer model, different ways of feeding I (cid:96) into it and different ways of converting transformer outputs into the CNN weights.

Original: Input samples were encoded by concatenating local and shared features from I (cid:96) to placeholder label embeddings ξ ( c ) with ξ : [1 , n ] → R d .
Revised: Input samples were encoded by concatenating local and shared features from I (cid:96) to trainable label embeddings ξ ( c ) with ξ : [1 , n ] → R d .

Original: Here n is the number of classes per episode and d is a chosen size of the label encoding.
Revised: Here n is the number of classes per episode and d is a chosen size of the label encoding.

Original: Note that the label embedding do not contain semantic information, but rather act as placeholder to differentiate different labels.
Revised: Note that the label embeddings do not contain semantic information, but rather act as placeholders to differentiate between distinct classes.

Original: Along with  input samples, the  transformer is also fed placeholder tokens for different slices of the weight tensors that the model needs to generate. Each placeholder is a learnable token representing a  slice of the  weight tensor 1 .
Revised: Along with the input samples, the sequence passed to the transformer was also populated with special learnable placeholder tokens 2 , each associated with a particular slice of the to-be-generated weight tensor.

Original: We used zero padding to make the ﬁnal input sample and weight slice tokens to have the same dimensions.

Original: After each weight placeholder was processed by the transformer, we read outthe corresponding model outputs  and assemble output weights (with ﬁrst d dimensions ignored) into the ﬁnal weight tensors.
Revised: After the entire input sequence was processed by the transformer, we read out model outputs associated with the weight slice placeholder tokens and assembled output weight slices into the ﬁnal weight tensors (see Fig. 2).

Original: In our experiments we also considered two different ways of encoding k × k × n input × n output convolutional kernels: (a) generating n output weight slices with each output token having a dimension of k 2 × n input (we call it “output allocation”), (b) kweight slices of size n input × n output (“spatial allocation”).
Revised: In our experiments we also considered two different ways of encoding k × k × n input × n output convolutional kernels: (a) generating n output weight slices with each output token having a dimension of k 2 × n input (we call it “output allocation”), (b) kweight slices of size n input × n output (“spatial allocation”).

Original: We show these results in Supplementary
Revised: We show these results in Supplementary

Original: Materials.
Revised: Materials.

Original: Transformer model.
Revised: Transformer model.

Original: The input discussed above was passed through a sequence of transformer encoder layers and the output weight tokens were then concatenated into a full weight tensor (see
Revised: The input discussed above was passed through a sequence of transformer encoder layers and the output weight tokens were then concatenated into a full weight tensor (see

Original: Fig. 2a).
Revised: Fig. 2).

Original: In one version of our model, we only used encoder layers, while in the other version we used both encoder and decoder on the same sequence of input tokens (see Fig. 2b). We compare these models in Supplementary Materials.
Revised: Experiments with alternative architectures employing transformer decoders are discussed in Supplementary Materials.

Original: Training the model.
Revised: Training the model.

Original: The weight generation model uses the support set to produce the weights of some or all CNN model layers.
Revised: The weight generation model uses the support set to produce the weights of some or all CNN model layers.

Original: This generated CNN model then processes the query set samples and the cross-entropy classiﬁcation loss is computed.
Revised: This generated CNN model then processes the query set samples and the cross-entropy classiﬁcation loss is computed.

Original: The weight generation parameters φ are then learned by optimizing this loss function using stochastic gradient descent.
Revised: The weight generation parameters φ are then learned by optimizing this loss function using stochastic gradient descent.

Original: 3. R EASONING B EHIND THE S ELF -A TTENTION M ECHANISM
Revised: 3. R EASONING B EHIND THE S ELF -A TTENTION M ECHANISM

Original: The choice of self-attention mechanism for the weight generator is not arbitrary.
Revised: The choice of self-attention mechanism for the weight generator is not arbitrary.

Original: One motivating reason behind this choice is that the output produced by generator with  self-attention is by design invariant to input permutations, i.e., permutations of samples in the training dataset.
Revised: One motivating reason behind this choice is that the output produced by generator with the basic self-attention is by design invariant to input permutations, i.e., permutations of samples in the training dataset.

Original: This also makes it suitable for processing unbalanced batches and batches with a variable number of samples (see Sec. 4.3).
Revised: This also makes it suitable for processing unbalanced batches and batches with a variable number of samples (see Sec. 4.3).

Original: Now we show that self-attention can express several intuitive algorithms, thus further motivating its utility.
Revised: Now we show that self-attention can express several intuitive algorithms, thus further motivating its utility.

Original: Supervised learning.
Revised: Supervised learning.

Original: Self-attention in its rudimentary form can implement a method similar to cosine-similarity-based sample weighting encoded in the logits layer 2 with weights W :
Revised: Self-attention in its rudimentary form can implement a method similar to cosine-similarity-based sample weighting encoded in the logits layer 3 with weights W :

Original: [Equation]
Revised: [Equation]

Original: which can also be viewed as a result of applying a single gradient descent step on the cross-entropy loss (see Appendix A).
Revised: which can also be viewed as a result of applying a single gradient descent step on the cross-entropy loss (see Appendix A).

Original: Here n is the total number of support-set samples { x ( m ) | m = 1 , . . . , n  } and e ( m ) , y ( m ) are the embedding vector and the one-hot label corresponding to x ( m ) .
Revised: Here n is the total number of support-set samples { x ( m ) | m ∈ [1 , n ] } and e ( m ) , y ( m ) are the embedding vector and the one-hot label corresponding to x ( m ) .

Original: The approach can be outlined (see more details in Appendix A) as follows.
Revised: The approach can be outlined (see more details in Appendix A) as follows.

Original: The self-attention operation receives encoded input samples
Revised: The self-attention operation receives encoded input samples

Original: I k = ( ξ ( c k ) , e k ) and weight placeholders ( µ ( i ) , 0) as its input.
Revised: I k = ( ξ ( c k ) , e k ) and weight placeholders ( µ ( i ) , 0) as its input.

Original: If each weight slice W i, · represented by a particular token ( µ ( i ) , 0) produces a query Q i that only attends to keys K k corresponding to samples I k with labels c k matching i and the values of these samples are set to their embeddings e k , then the self-attention operation will essentially average the embeddings of all samples assigned label i thus matching the ﬁrst term in W in equation 1.
Revised: If each weight slice W i, · represented by a particular token ( µ ( i ) , 0) produces a query Q i that only attends to keys K k corresponding to samples I k with labels c k matching i and the values of these samples are set to their embeddings e k , then the self-attention operation will essentially average the embeddings of all samples assigned label i thus matching the ﬁrst term in W in equation 1.

Original: Semi-supervised learning.
Revised: Semi-supervised learning.

Original: A similar self-attention mechanism can also be designed to produce logits layer weights when the support set contains some unlabeled samples.
Revised: A similar self-attention mechanism can also be designed to produce logits layer weights when the support set contains some unlabeled samples.

Original: The mechanism ﬁrst propagates classes of labeled samples to similar unlabeled samples.
Revised: The mechanism ﬁrst propagates classes of labeled samples to similar unlabeled samples.

Original: This can be achieved by choosing the queries and the keys of the samples to be proportional to their embeddings.
Revised: This can be achieved by choosing the queries and the keys of the samples to be proportional to their embeddings.

Original: The attention map for sample i would then be deﬁned by a softmax of e i · e j , or in other words would be proportional to exp( e i · e j ) .
Revised: The attention map for sample i would then be deﬁned by a softmax of e i · e j , or in other words would be proportional to exp( e i · e j ) .

Original: Choosing sample values to be proportional to the class tokens, we can then propagate a class of a labeled sample e j to a nearby unlabeled sample with embedding e i , for which e i · e j is sufﬁciently large.
Revised: Choosing sample values to be proportional to the class tokens, we can then propagate a class of a labeled sample e j to a nearby unlabeled sample with embedding e i , for which e i · e j is sufﬁciently large.

Original: If the self-attention module is “residual”, i.e., the output of the self-attention operation is added to the original input, like it is done in the transformer model, then this additive update would essentially “mark” an unlabeled sample by the propagated class (albeit this term might have a small norm).
Revised: If the self-attention module is “residual”, i.e., the output of the self-attention operation is added to the original input, like it is done in the transformer model, then this additive update would essentially “mark” an unlabeled sample by the propagated class (albeit this term might have a small norm).

Original: The second self-attention layer can then be designed similarly to the supervised case.
Revised: The second self-attention layer can then be designed similarly to the supervised case.

Original: If label embeddings are orthogonal, then even a small component of a class label can be sufﬁcient for a weight slice to attend to it thus adding its embedding to the ﬁnal weight.
Revised: If label embeddings are orthogonal, then even a small component of a class label can be sufﬁcient for a weight slice to attend to it thus adding its embedding to the ﬁnal weight.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present H YPER T RANSFORMER (HT) experimental results and discuss the implications of our empirical ﬁndings.
Revised: In this section, we present H YPER T RANSFORMER (HT) experimental results and discuss the implications of our empirical ﬁndings.

Original: 4. D ATASETS AND S ETUP
Revised: 4. D ATASETS AND S ETUP

Original: Datasets.
Revised: Datasets.

Original: For our experiments, we chose several most widely used few-shot datasets including
Revised: For our experiments, we chose several most widely used few-shot datasets including

Original: O MNIGLOT , MINI I MAGE N ET and TIERED I MAGE N ET .
Revised: O MNIGLOT , MINI I MAGE N ET and TIERED I MAGE N ET .

Original: MINI I MAGE N ET contains a relatively small set of labels and is arguably the simplest to overﬁt to.
Revised: MINI I MAGE N ET contains a relatively small set of labels and is arguably the simplest to overﬁt to.

Original: Because of this and since in many recent publications MINI I MAGE N ET was replaced with a more realistic TIERED I MAGE N ET datasets, we conduct many of our experiments and ablation studies using O MNIGLOT and TIERED I MAGE N ET datasets.
Revised: Because of this and since in many recent publications MINI I MAGE N ET was replaced with a more realistic TIERED I MAGE N ET datasets, we conduct many of our experiments and ablation studies using O MNIGLOT and TIERED I MAGE N ET datasets.

Original: Models.
Revised: Models.

Original: While HT can support generation of arbitrarily large weight tensors by ﬂattening and slicing the entire weight tensor, in this work, we limit our experiments to HT models that generate slices encoding individual output channels directly.
Revised: While HT can support generation of arbitrarily large weight tensors by ﬂattening and slicing the entire weight tensor, in this work, we limit our experiments to HT models that generate slices encoding individual output channels directly.

Original: For the target models we focus on 4-layer architectures identical to those used in MAML++ and numerous other papers.
Revised: For the target models we focus on 4-layer architectures identical to those used in MAML++ and numerous other papers.

Original: Generating larger architectures such as R ES N ET and W IDE R ES N ET will be the subject of our future work.
Revised: Generating larger architectures such as R ES N ET and W IDE R ES N ET will be the subject of our future work.

Original: More speciﬁcally, we used a sequence of four 3 × 3 convolutional layers with the same number of output channels followed by batch normalization layers, nonlinearities and max-pooling stride-2 layers.
Revised: More speciﬁcally, we used a sequence of four 3 × 3 convolutional layers with the same number of output channels followed by batch normalization layers, nonlinearities and max-pooling stride-2 layers.

Original: All
Revised: All

Original: BN variables were learned and not generated 3 .
Revised: BN variables were learned and not generated 4 .

Original: [Table]
Revised: [Table]

Original: 4. S UPERVISED R ESULTS WITH L OGITS L AYER G ENERATION
Revised: S UPERVISED R ESULTS WITH L OGITS L AYER G ENERATION

Original: Our ﬁrst experiments compared the proposed HT approach with MAML++ on O MNIGLOT , MINII MAGE N ET and TIERED I MAGE N ET datasets (see Table 1).
Revised: Our ﬁrst experiments compared the proposed HT approach with MAML++ on O MNIGLOT , MINII MAGE N ET and TIERED I MAGE N ET datasets (see Table 1).

Original: Interestingly, for models that had more than 8 channels per layer, the results obtained with HT generating the ﬁnal logits layer proved to be nearly identical to those where HT was used to generate all CNN layers (see Section 4.4).
Revised: Interestingly, for models that had more than 8 channels per layer, the results obtained with HT generating the ﬁnal logits layer proved to be nearly identical to those where HT was used to generate all CNN layers (see Section 4.4).

Original: In our experiments the number of local features was chosen to be the same as the number of model channels and the shared feature had a dimension of 32 regardless of the model size.
Revised: In our experiments the number of local features was chosen to be the same as the number of model channels and the shared feature had a dimension of 32 regardless of the model size.

Original: The shared feature extractor was a simple 4-layer convolutional model with batch normalization and stride- 2 3 ×convolutional kernels.
Revised: The shared feature extractor was a simple 4-layer convolutional model with batch normalization and stride- 2 3 ×convolutional kernels.

Original: Local feature extractors were two-layer convolutional models with outputs of both layers averaged over the spatial dimensions and concatenated to produce the ﬁnal local feature.
Revised: Local feature extractors were two-layer convolutional models with outputs of both layers averaged over the spatial dimensions and concatenated to produce the ﬁnal local feature.

Original: For all tasks except 5-shot MINI I MAGE N ET our transformer had 3 layers, used a simple sequence of encoder layers (Figure 2b-i) and used the “output allocation” of weight slices (Section 3.1).
Revised: For all tasks except 5-shot MINI I MAGE N ET our transformer had 3 layers, used a simple sequence of encoder layers (Figure 2b-i) and used the “output allocation” of weight slices (Section 3.1).

Original: Experiments with the encoder-decoder transformer architecture can be found in Appendix C.
Revised: Experiments with the encoder-decoder transformer architecture can be found in Appendix D.

Original: The 5-shot
Revised: The 5-shot

Original: MINI I MAGE N ET results presented in Table 1 were obtained with a simpliﬁed transformer model that had 1 layer, and did not have the ﬁnal fully-connected layer and nonlinearity.
Revised: MINI I MAGE N ET results presented in Table 1 were obtained with a simpliﬁed transformer model that had 1 layer, and did not have the ﬁnal fully-connected layer and nonlinearity.

Original: This proved necessary for reducing model overﬁtting of this smaller dataset.
Revised: This proved necessary for reducing model overﬁtting of this smaller dataset.

Original: Other model parameters are described in detail in Appendix B.
Revised: Other model parameters are described in detail in Appendix B.

Original: Results obtained with our method in a few-shot setting (see Table 1) are frequently better than
Revised: Results obtained with our method in a few-shot setting (see Table 1) are frequently better than

Original: MAML++ and P ROTO N ET results and are comparable to those obtained with much more advanced methods (see Table 2). The improved performance on smaller models canprobably be attributed to parameter disentanglement between the weight generator and the CNN model. 
Revised: MAML++ results, especially on smaller models, which can be attributed to parameter disentanglement between the weight generator and the CNN model. While the improvement over MAML++ gets smaller with the growing size of the generated CNN, our results on MINI I MAGE N ET and

Original:  While the learned HT model could perform a relatively simple calculation on high-dimensional sample features, perhaps not too different from that in equation 1, our brief analysis of the parameters space (see Appendix C) shows that using simpler 1-layer transformers leads to a modest decrease of the test accuracy and a greater drop in the training accuracy for smaller models.
Revised: TIERED I MAGE N ET appear to be comparable to those obtained with numerous other advanced methods (see Table 2). Discussion of additional comparisons to LGM-Net (Li et al., 2019b) and LEO (Rusu et al., 2019) using a different setup (which is why they could not be included in Table 2) and showing an almost identical 1-shot performance can be found in Appendix C. While the learned HT model could perform a relatively simple calculation on high-dimensional sample features, perhaps not too different from that in equation 1, our brief analysis of the parameters space (see Appendix D) shows that using simpler 1-layer transformers leads to a modest decrease of the test accuracy and a greater drop in the training accuracy for smaller models.

Original: We observed that the results in Tablecould be improved even further by increasing the feature sizes (see Appendix C), but we did not pursue an exhaustive optimization in the parameter space.
Revised: We observed that the results in Tablecould be improved even further by increasing the feature sizes (see Appendix D), but we did not pursue an exhaustive optimization in the parameter space.

Original: It is worth noting that overﬁtting leading to a good performance on tasks composed of seen categories, but poor generalization to unseen categories, may still have practical applications.
Revised: It is worth noting that overﬁtting leading to a good performance on tasks composed of seen categories, but poor generalization to unseen categories, may still have practical applications.

Original: Speciﬁcally, if the actual task relies on classes seen at the training time, we can generate a model customized to a particular task in a single pass without having to perform any SGD steps to ﬁne-tune the model.
Revised: Speciﬁcally, if the actual task relies on classes seen at the training time, we can generate a model customized to a particular task in a single pass without having to perform any SGD steps to ﬁne-tune the model.

Original: This is useful if, for example, the client model needs to be adjusted to a particular set of known classes most widely used by this client.
Revised: This is useful if, for example, the client model needs to be adjusted to a particular set of known classes most widely used by this client.

Original: We also anticipate that with more complex data augmen- tations and additional synthetic tasks, more complex transformer-based models can further improve their performance on the test set and a deeper analysis of such techniques will be the subject of our future work. 
Revised: We also anticipate that with more complex data augmentations and additional synthetic tasks, more complex transformer-based models can further improve their performance on the test set and a deeper analysis of such techniques will be the subject of our future work. [Table]

Original: [Table]
Revised: [Table]

Original: 4. S EMI -S UPERVISED R ESULTS WITH L OGITS L AYER G ENERATION
Revised: AYER G ENERATION

Original: In our approach, the weight generation model is trained by optimizing the loss calculated on the query set and therefore any additional information about the task, including unlabeled samples, can be provided as a part of the support set to the weight generator without having to alter the optimization objective.
Revised: In our approach, the weight generation model is trained by optimizing the loss calculated on the query set and therefore any additional information about the task, including unlabeled samples, can be provided as a part of the support set to the weight generator without having to alter the optimization objective.

Original: This allows us to tackle a semi-supervised few-shot learning problem without making any substantial changes to the model or the training approach.
Revised: This allows us to tackle a semi-supervised few-shot learning problem without making any substantial changes to the model or the training approach.

Original: In our implementation, we simply added unlabeled samples into the support set and marked them with an auxiliary learned “unlabeled” token ˆ ξ in place of the label encoding ξ ( c ) .
Revised: In our implementation, we simply added unlabeled samples into the support set and marked them with an auxiliary learned “unlabeled” token ˆ ξ in place of the label encoding ξ ( c ) .

Original: Since O MNIGLOT is typically characterized by very high accuracies in the 97% − 99% range, we conducted all our experiments with TIERED I MAGE N ET .
Revised: Since O MNIGLOT is typically characterized by very high accuracies in the 97% − 99% range, we conducted all our experiments with TIERED I MAGE N ET .

Original: Results of our experiments presented in
Revised: Results of our experiments presented in

Original: Table 3 show that adding unlabeled samples leads to a substantial increase of the ﬁnal test accuracy.
Revised: Table 3 show that adding unlabeled samples leads to a substantial increase of the ﬁnal test accuracy.

Original: Furthermore, notice that the model achieves its best performance when the number of transformer layers is greater than one.
Revised: Furthermore, notice that the model achieves its best performance when the number of transformer layers is greater than one.

Original: This is consistent with the basic mechanism discussed in Section 3.2 and requiring two self-attention layers to function.
Revised: This is consistent with the basic mechanism discussed in Section 3.2 and requiring two self-attention layers to function.

Original: It is worth noticing that adding more unlabeled samples into the support set makes our model more difﬁcult to train and it gets stuck producing CNNs with essentially random outputs.
Revised: It is worth noticing that adding more unlabeled samples into the support set makes our model more difﬁcult to train and it gets stuck producing CNNs with essentially random outputs.

Original: Our solution was to introduce unlabeled samples incrementally during training.
Revised: Our solution was to introduce unlabeled samples incrementally during training.

Original: This was implemented by masking out some unlabeled samples in the beginning of the training and then slowly reducing the masking probability over time.
Revised: This was implemented by masking out some unlabeled samples in the beginning of the training and then gradually reducing the masking probability over time.

Original: 4. G ENERATING M ORE M ODEL L AYERS
Revised: 4. G ENERATING M ORE M ODEL L AYERS

Original: We demonstrated that HT model can outperform MAML++ on common few-shot learning datasets by generating just the last logits layer of the CNN model.
Revised: We demonstrated that HT model can outperform MAML++ on common few-shot learning datasets by generating just the last logits layer of the CNN model.

Original: But under what conditions can it be advantageous to generate additional CNN layers?
Revised: But under what conditions can it be advantageous to generate additional CNN layers?

Original: As we show here, generating all CNN layers with a multi-layer transformer can lead to a signiﬁcant performance improvement on CNN models below a particular size.
Revised: As we show here, generating all CNN layers with a multi-layer transformer can lead to a signiﬁcant performance improvement on CNN models below a particular size.

Original: We demonstrated this by conducting experiments, in which some of the layers were generated and some layers were learned (usually ﬁrst few layers of the CNN).
Revised: model uses L T encoder layers.

Original: For O MNIGLOT dataset, we saw that both training and test accuracies for a 4 -channel and 6 -channel CNNs increased with the number of generated layers (see Fig. 3 and Table 4 in Appendix) and using more complex transformer models
Revised: Notice a performance improvement of semi-supervised learning over the 1-shot supervised results. Accuracy is seen to grow with the number of unlabeled samples and the maximum accuracy is reached when the encoder has at least two layers.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: We demonstrated this by conducting experiments, in which some layers were generated and some layers were learned (usually the ﬁrst few layers of the CNN).

Revised: For O MNIGLOT dataset, we saw that both training and test accuracies for a 4-channel and a 6-channel CNNs increased with the number of generated layers (see Fig.

Original:  with 2 or more encoder layers improved both training and test accuracies of fully-generated CNN models of this size (see Appendix C).
Revised: 3 and Table 4 in Appendix) and using more complex transformer models with 2 or more encoder layers improved both training and test accuracies of fully-generated CNN models of this size (see Appendix D).

Original: However, as the size of the model increased and reachedchannels, generating the last logits layer alone proved to be sufﬁcient for getting the best results on
Revised: However, as the size of the model increased and reachedchannels, generating the last logits layer alone proved to be sufﬁcient for getting the best results on

Original: O MNIGLOT and TIERED
Revised: O MNIGLOT and TIERED

Original: I MAGE N ET .
Revised: I MAGE N ET .

Original: The positive effect of generating convolutional layers can also be observed in shallow models with large convolutional kernels with large strides where the model performance can be much more sensitive to a proper choice of model weights.
Revised: The positive effect of generating convolutional layers can also be observed in shallow models with large convolutional kernels with large strides where the model performance can be much more sensitive to a proper choice of model weights.

Original: For example, in a 16-channel model with two convolutional kernels of size 9 and the stride of 4 , the overall test accuracy for a model generating only the ﬁnal convolutional layer was about 1% lower than the accuracies of the models generating at least one additional convolutional ﬁlter.
Revised: For example, in a 16-channel model with two convolutional kernels of size 9 and the stride of 4 , the overall test accuracy for a model generating only the ﬁnal convolutional layer was about 1% lower than the accuracies of the models generating at least one additional convolutional ﬁlter.

Original: We also speculate, that as the complexity of the task increases, generating some or all intermediate network layers should become more important for achieving optimal performance.
Revised: We also speculate that as the complexity of the task increases, generating some or all intermediate network layers should become more important for achieving optimal performance.

Original: Verifying this hypothesis and understanding the “boundary” in the model space between two regimes where a static backbone is sufﬁcient, or no longer sufﬁcient will be the subject of our future work.
Revised: Verifying this hypothesis and understanding the “boundary” in the model space between two regimes where a static backbone is sufﬁcient, or no longer sufﬁcient will be the subject of our future work.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, we proposed a HyperTransformer (HT), a novel transformer-based model that generates all weights of a CNN model directly from a few-shot support set.
Revised: In this work, we proposed a HyperTransformer (HT), a novel transformer-based model that generates all weights of a CNN model directly from a few-shot support set.

Original: This approach allows  to use a high-capacity model for encoding task-dependent variations in the weights of a smaller model.
Revised: This approach allows us to use a high-capacity model for encoding task-dependent variations in the weights of a smaller model.

Original: We demonstrate that generating the last logits layer alone, the transformer-based weight generator beats or matches performance of multiple traditional learning methods on several few-shot benchmarks.
Revised: We demonstrate that generating the last logits layer alone, the transformer-based weight generator beats or matches performance of multiple traditional learning methods on several few-shot benchmarks.

Original: More importantly, we showed that HT can be straightforwardly extended to handle unlabeled samples that might be present in the support set and our experiments demonstrate a considerable fewshot performance improvement in the presence of unlabeled data.
Revised: More importantly, we showed that HT can be straightforwardly extended to handle unlabeled samples that might be present in the support set and our experiments demonstrate a considerable fewshot performance improvement in the presence of unlabeled data.

Original: Finally, we explore the impact of the transformer-encoded model diversity in CNN models of different sizes.
Revised: Finally, we explore the impact of the transformer-encoded model diversity in CNN models of different sizes.

Original: We use HT to generate some or all convolutional kernels and biases and show that for sufﬁciently small models, adjusting all model parameters further improves their few-shot learning performance.
Revised: We use HT to generate some or all convolutional kernels and biases and show that for sufﬁciently small models, adjusting all model parameters further improves their few-shot learning performance.