Original: Sampling from Hyper-Representations to Generate Neural Network Weights
Revised: Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Learning representations of neural network weights given a model zoo is an emerging and challenging area with many potential applications from model inspection,to neural architecture search or knowledge distillation.
Revised: Learning representations of neural network weights given a model zoo is an emerging and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation.

Original: Recently, an autoencodertrained on a model zoo was able to learn a hyper-representation , which capturesintrinsic and extrinsic properties of the models in the zoo.
Revised: Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation , which captures intrinsic and extrinsic properties of the models in the zoo.

Original: In this work, we extend hyper-representations for generative use to sample new model weights.
Revised: In this work, we extend hyper-representations for generative use to sample new model weights.

Original: Wepropose layer-wise loss normalization which we demonstrate is key to generatehigh-performing models and several sampling methods based on the topology ofhyper-representations.
Revised: We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and several sampling methods based on the topology of hyper-representations.

Original: The models generated using our methods are diverse, performant and capable to outperform strong baselines as evaluated on several downstream tasks: initialization, ensemble sampling and transfer learning.
Revised: The models generated using our methods are diverse, performant and capable to outperform strong baselines as evaluated on several downstream tasks: initialization, ensemble sampling and transfer learning.

Original: Our resultsindicate the potential of knowledge aggregation from model zoos to new modelsvia hyper-representations thereby paving the avenue for novel research directions.
Revised: Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.

Original:  Introduction
Revised:  Introduction

Original: Over the last decade, countless neural network models have been trained and uploaded to differentmodel hubs.
Revised: Over the last decade, countless neural network models have been trained and uploaded to different model hubs.

Original: Many factors such as random initialization and no global optimum ensure that thetrained models are different from one another.
Revised: Many factors such as random initialization and no global optimum ensure that the trained models are different from one another.

Original: What could we learn from such a population of neuralnetwork models?
Revised: What could we learn from such a population of neural network models?

Original: Since the parameter space of neural networks is complex and high-dimensional,representation learning from such populations (often referred to as model zoos) has become anemerging and challenging area.
Revised: Since the parameter space of neural networks is complex and high-dimensional, representation learning from such populations (often referred to as model zoos) has become an emerging and challenging area.

Original: Recent work along that direction has demonstrated the ability of such learned representations tocapture intrinsic and extrinsic properties of the models in a zoo [36, 33, 25].
Revised: Recent work along that direction has demonstrated the ability of such learned representations to capture intrinsic and extrinsic properties of the models in a zoo [40, 37, 27].

Original: According to [33], NNspopulate a low dimensional manifold, which can be learned with an autoencoder via self-supervisedlearning directly from the weights 1 without access to the original image data and labels.
Revised: According to [37], NNs populate a low dimensional manifold, which can be learned with an autoencoder via self-supervised learning directly from the model paramters (weights and biases) without access to the original image data and labels.

Original: This socalled hyper-representation has been demonstrated to be useful to predict several model propertiessuch as accuracy, hyperparameters or architecture conﬁgurations.
Revised: This so called hyper-representation has been demonstrated to be useful to predict several model properties such as accuracy, hyperparameters or architecture configurations.

Original: However, [33] focused on discriminative downstream tasks by exploiting the encoder only.
Revised: However, [37] focused on discriminative downstream tasks by exploiting the encoder only.

Original: We takeone step further and extend their work towards the generative downstream tasks by sampling modelweights directly from the task-agnostic hyper-representation.
Revised: We take one step further and extend their work towards the generative downstream tasks by sampling model weights directly from the task-agnostic hyper-representation.

Original: To that end, we introduce a layer-wisenormalization that improves the quality of decoded neural network weights signiﬁcantly.
Revised: To that end, we introduce a layer-wise normalization that improves the quality of decoded neural network weights significantly.

Original: Based on acareful analysis of the geometry, smoothness and robustness of this space, we also propose severalsampling methods to generate weights in a single forward pass from the hyper-representation.
Revised: Based on a careful analysis of the geometry, smoothness and robustness of this space, we also propose several sampling methods to generate weights in a single forward pass from the hyper-representation.

Original: Weevaluate our approach on four image datasets and three generative downstream tasks of (i) modelSubmitted Do not distribute. initialization, (ii) ensemble sampling, and (iii) transfer learning.
Revised: We evaluate our approach on four image datasets and three generative downstream tasks of (i) model initialization, (ii) ensemble sampling, and (iii) transfer learning.

Original: Our results demonstrate its capabilityto out-perform previous hyper-representation learning and conventional baselines.
Revised: Our results demonstrate its capability to out-perform previous hyper-representation learning and conventional baselines.

Original: Previous work on generating model weights proposed (Graph) HyperNetworks [13, 39, 18], Bayesian
Revised: Previous work on generating model weights proposed (Graph) HyperNetworks [14, 45, 21], Bayesian

Original: HyperNetworks [7], HyperGANs [31] and HyperTransformers [40] for neural architecture search,model compression, ensembling, transfer- or meta-learning.
Revised: HyperNetworks [8], HyperGANs [35] and HyperTransformers [46] for neural architecture search, model compression, ensembling, transfer- or meta-learning.

Original: These methods learn representations byusing images and labels of the target domain.
Revised: These methods learn representations by using images and labels of the target domain.

Original: In contrast, our approach only uses model weights anddoes not need access to underlying data samples and labels – an emergent use case, e.g. of deep learning monitoring services or model hubs.
Revised: In contrast, our approach only uses model weights and does not need access to underlying data samples and labels – an emergent use case, e.g. of deep learning monitoring services or model hubs.

Original: In addition to the ability to generate novel and diverse modelweights, compared to previous works our approach (a) can generate novel weights conditionally onmodel zoos from unseen tasks and (b) can be conditioned on the latent factors of the underlying hyperrepresentation.
Revised: In addition to the ability to generate novel and diverse model weights, compared to previous works our approach (a) can generate novel weights conditionally on model zoos from unseen tasks and (b) can be conditioned on the latent factors of the underlying hyperrepresentation.

Original: Notably, both (a) and (b) can be done without the need to retrain hyper-representations.
Revised: Notably, both (a) and (b) can be done without the need to retrain hyper-representations.

Original: The results suggest our approach (Figure 1) to be a promising step towards a general purposehyper-representation encapsulating knowledge of model zoos to advance different downstream tasks.
Revised: The results suggest our approach (Figure 1) to be a promising step towards a general purpose hyper-representation encapsulating knowledge of model zoos to advance different downstream tasks.

Original: The hyper-representations and code to reproduce our results will be made available upon publication.
Revised: The hyper-representations and code to reproduce our results are available at https://github.com/HSG-AIML/NeurIPS_2022-Generative_Hyper_Representations .

Original:  Background: Training Hyper-Representations
Revised:  Background: Training Hyper-Representations

Original: We summarize the ﬁrst stage of our method that corresponds to learning a hyper-representation of apopulation of neural networks, called a model zoo [33].
Revised: We summarize the first stage of our method that corresponds to learning a hyper-representation of a population of neural networks, called a model zoo [37].

Original: In [33] and this paper, a model zoo consistsof models trained on the same task such as CIFAR-10 image classiﬁcation [20].
Revised: In [37] and this paper, a model zoo consists of models trained on the same task such as CIFAR-10 image classification [23].

Original: Speciﬁcally, ahyper-representation is learned using an autoencoder ˆ w i
Revised: Specifically, a hyper-representation is learned using an autoencoder ˆ w i

Original: = h ( g ( w i )) on a zoo of M models { w i } M 1 ,where w i is the ﬂattened vector of dimension N of all the weights of the i -th model.
Revised: = h ( g ( w i )) on a zoo of M models { w i } M 1 , where w i is the flattened vector of dimension N of all the weights of the i -th model.

Original: The encoder gcompresses vector w i to ﬁxed-size hyper-representation zi
Revised: The encoder g compresses vector w i to fixed-size hyper-representation z

Original:  = g ( w i ) of lower dimension.
Revised: i = g ( w i ) of lower dimension.

Original: The decoderh decompresses the hyper-representation to the reconstructed vector
Revised: The decoder h decompresses the hyper-representation to the reconstructed vector

Original: ˆ w i .
Revised: ˆ w i .

Original: Both encoder and decoderare built on a self-attention blocks.
Revised: Both encoder and decoder are built on a self-attention block [41].

Original: The samples from model zoos are understood as sequences ofconvolutional or fully connected neurons.
Revised: The samples from model zoos are understood as sequences of convolutional or fully connected neurons.

Original: Each of the these is encoded as a token embedding andconcatenated to  a sequence.
Revised: Each of the neurons is encoded as a token embedding and concatenated to form a sequence.

Original: The sequence is passed through several layers of multi-head self-attention.
Revised: The sequence is passed through several layers of multi-head self-attention.

Original: Afterwards, a special compression token summarizing the entire sequence is linearly compressedto the bottleneck.
Revised: Afterwards, a special compression token summarizing the entire sequence is linearly compressed to the bottleneck.

Original: The output is fed through a tanh-activation to achieve a bounded latent space z ifor the hyper-representation.
Revised: The output is fed through a tanh-activation to achieve a bounded latent space z i for the hyper-representation.

Original: The decoder is symmetric to the encoder, the embeddings are linearlydecompressed from hyper-representations z i and position encodings  added.
Revised: The decoder is symmetric to the encoder, the embeddings are linearly decompressed from hyper-representations z i and position encodings are added.

Original: Training is done in a multi-task fashion, minimizing the composite loss L =  L MSE + (1   )
Revised: Training is done in a multi-objective fashion, minimizing the composite loss L = β L MSE +(1 − β )

Original: L c ,63 where L c is a contrastive loss and L MSE is a weight reconstruction loss (see details in [33]).
Revised: L c , where L c is a contrastive loss and L MSE is a weight reconstruction loss (see details in [37]).

Original: Wecan write the latter in a layer-wise way to facilitate our discussion in § 3.1:
Revised: We can write the latter in a layer-wise way to facilitate our discussion in § 3.1:

Original: [Equation]
Revised: [Equation]

Original: where ˆ w ( l ) i , w ( l ) i are reconstructed and original weights for the l -th layer of the i -th model in the zoo.
Revised: where ˆ w ( l ) i , w ( l ) i are reconstructed and original weights for the l -th layer of the i -th model in the zoo.

Original: The contrastive loss L c leverages two types of data augmentation at train time to impose structure onthe latent space: permutation exploiting 
Revised: The contrastive loss L c leverages two types of data augmentation at train time to impose structure on the latent space: permutation exploiting inherent symmetries of the weight space and random erasing.

Original:  Methods
Revised:  Methods

Original: In the following, we present (i) layer-wise loss normalization to ensure that decoded models areperformant, and (ii) sampling methods to generate diverse populations of models.
Revised: In the following, we present (i) layer-wise loss normalization to ensure that decoded models are performant, and (ii) sampling methods to generate diverse populations of models.

Original: 3.1 Layer-Wise Loss Normalization
Revised: 3.1 Layer-Wise Loss Normalization

Original: We observed that hyper-representations as proposed by [33] decode to dysfunctional models, with performance around random guessing.
Revised: We observed that hyper-representations as proposed by [37] decode to dysfunctional models, with performance around random guessing.

Original: To alleviate that, we propose a novel layer-wise loss normalization(LWLN), which we motivate and detail in the following.
Revised: To alleviate that, we propose a novel layer-wise loss normalization (LWLN), which we motivate and detail in the following.

Original: Due to the MSE training loss in (1), the reconstruction error can generally be expected to beuniformly distributed over all weights and layers of the weight vector w .
Revised: Due to the MSE training loss in (1), the reconstruction error can generally be expected to be uniformly distributed over all weights and layers of the weight vector w .

Original: However, the weightmagnitudes of many of our zoos are unevenly distributed across different layers.
Revised: However, the weight magnitudes of many of our zoos are unevenly distributed across different layers.

Original: In these zoos, theeven distribution of reconstruction errors lead to undesired effects.
Revised: In these zoos, the even distribution of reconstruction errors lead to undesired effects.

Original: Layers with broader distributionsand large weights are reconstructed well, while layers with narrow distributions and small weights aredisregarded.
Revised: Layers with broader distributions and large-magnitude weights are reconstructed well, while layers with narrow distributions and small-magnitude weights are disregarded.

Original: These layers can become a weak link in the reconstructed models, causing performanceto drop signiﬁcantly down to random guessing.
Revised: The latter layers can become a weak link in the reconstructed models, causing performance to drop significantly down to random guessing.

Original: The top row of Figure 2 shows an example of abaseline hyper-representation learned on the zoo of SVHN models [29].
Revised: The top row of Figure 2 shows an example of a baseline hyper-representation learned on the zoo of SVHN models [32].

Original: Common initializationschemes [14, 10] produce distributions with different scaling factors per layer, so the issue is notan artifact of the zoos, but can exist in real world model populations.
Revised: Common initialization schemes [15, 11] produce distributions with different scaling factors per layer, so the issue is not an artifact of the zoos, but can exist in real world model populations.

Original: Similarly, recent work ongenerating models normalizes weights to boost performance [18].
Revised: Similarly, recent work on generating models normalizes weights to boost performance [21].

Original: In order to achieve equallyaccurate reconstruction across the layers, we introduce a layer-wise loss normalization (LWLN) withthe mean µ l and standard deviation  l of all weights in layer l estimated over the train split of the zoo: 
Revised: In order to achieve equally accurate reconstruction across the layers, we introduce a layer-wise loss normalization (LWLN) with the mean µ l and standard deviation σ l of all weights in layer l estimated over the train split of the zoo: L ¯ MSE = MN M (cid:88) i =

Revised: L (cid:88) l = (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)

Original: [Equation]
Revised: ˆ w ( l ) i − µ l σ l − w ( l ) i − µ l σ l (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)  = MN M (cid:88) i = L (cid:88) l = ∥ ˆ w ( l ) i − w ( l ) i ∥σ 2 l .

Original: 3.2 Sampling from Hyper-Representations
Revised: 3.2 Sampling from Hyper-Representations

Original: We introduce methods to draw diverse and high-quality samples z ⇤ ⇠ p ( z ) from the learned hyperrepresentation space to generate model weights w ⇤ =
Revised: We introduce methods to draw diverse and high-quality samples z ∗ ∼ p ( z ) from the learned hyperrepresentation space to generate model weights w ∗ =

Original: h ( z ⇤ ) .
Revised: h ( z ∗ ) .

Original: Such sampling is facilitated if there isknowledge on the topology of the space spanned by z .
Revised: Such sampling is facilitated if there is knowledge on the topology of the space spanned by z .

Original: One way to achieve that is to train a variationalautoencoder (VAE) with a predeﬁned prior [17] instead of the autoencoder of [33].
Revised: One way to achieve that is to train a variational autoencoder (VAE) with a predefined prior [20] instead of the autoencoder of [37].

Original: While training VAEs on common domains such as images has become well-understood and feasible, in our relativelynovel weights domain, we found it problematic (see details in Appendix E).
Revised: While training VAEs on common domains such as images has become well-understood and feasible, in our relatively novel weight domain, we found it problematic (see details in Appendix E).

Original: Other  methods avoid apredeﬁned prior of VAEs, either by analyzing the topology of the space learned by the autoencoderor ﬁtting a separate density estimation model on top of the learned representation [24, 12].
Revised: Other generative methods avoid a predefined prior of VAEs, either by analyzing the topology of the space learned by the autoencoder or fitting a separate density estimation model on top of the learned representation [26, 13].

Original: Thesemethods assume the representation space to have strong regularities.
Revised: These methods assume the representation space to have strong regularities.

Original: The hyper-representationspace learned by the autoencoder of [33] is already regularized by dropout regularization applied tothe encoder and decoder as in [9].
Revised: The hyper-representation space learned by the autoencoder of [37] is already regularized by dropout regularization applied to the encoder and decoder as in [10].

Original: The contrastive loss component requiring similar models to beembedded close to each other may also improve the regularity of the representation space.
Revised: The contrastive loss component requiring similar models to be embedded close to each other may also improve the regularity of the representation space.

Original: Empirically, we found our layer-wise loss normalization (LWLN) to further regularize the representation space byensuring robustness and smoothness (see Figure 3 in § 4).
Revised: Empirically, we found our layer-wise loss normalization (LWLN) to further regularize the representation space by ensuring robustness and smoothness (see Figure 3 in § 4).

Original: Given the smoothness and robustness of the learned hyper-representation space, we follow [24, 12, 9]in estimating the density and topology to draw samples from a regularized autoencoder.
Revised: Given the smoothness and robustness of the learned hyper-representation space, we follow [26, 13, 10] in estimating the density and topology to draw samples from a regularized autoencoder.

Original: To that end,we introduce three strategies to sample from that space: S KDE , S Neigh , S GAN .
Revised: To that end, we introduce three strategies to sample from that space: S KDE , S Neigh , S GAN .

Original: To model the density andtopology in representation space, we use the embeddings of the train set as anchor samples { z i } .
Revised: To model the density and topology in representation space, we use the embeddings of the train set as anchor samples { z i } .

Original: Weobserve that many anchor samples from { z i } correspond to the models with relatively poor accuracy(Figure 2), so to improve the quality of sampled weights, we consider the variants of these methods using only those embeddings of training samples corresponding to the top 30% performing models.
Revised: We observe that many anchor samples from { z i } correspond to the models with relatively poor accuracy (Figure 2), so to improve the quality of sampled weights, we consider the variants of these methods using only those embeddings of training samples corresponding to the top 30% performing models.

Original: Wedenote these sampling methods as S KDE30 , S Neigh30 , S GAN30 respectively.
Revised: We denote these sampling methods as S KDE30 , S Neigh30 , S GAN30 respectively.

Original: These methods can potentially decrease sample diversity, however, we found that the generated weights are still diverse enough(e.g. to construct high-performant ensembles, Figure 5).
Revised: These methods can potentially decrease sample diversity, however, we found that the generated weights are still diverse enough (e.g. to construct high-performant ensembles, Figure 5).

Original: Finally, as baseline and sanity check weexplore sampling uniformly in representation space S U and sampling in low-probability regions S C .
Revised: Finally, as baseline and sanity check we explore sampling uniformly in representation space S U and sampling in low-probability regions S C .

Original: 3.2.1 Uniform S U
Revised: 3.2.1 Uniform S U

Original: As a naive baseline, we draw samples uniformly in hyper-representation space (bounded by tanh, § 2)and denote it as S U .
Revised: As a naive baseline, we draw samples uniformly in hyper-representation space (bounded by tanh, § 2) and denote it as S U .

Original: This is naive, because we found that the embeddings z populate only sections of ashell of a high-dimensional sphere (see Figures10 and 11  in Appendix D).
Revised: This is naive, because we found that the embeddings z populate only sections of a shell of a high-dimensional sphere (see Figures 11 and 12 in Appendix D).

Original: So most of the uniform samples lie in the low-probability regions of the space and are not expected to be decoded to useful models.
Revised: So most of the uniform samples lie in the low-probability regions of the space and are not expected to be decoded to useful models.

Original: 3.2.2 Density estimation S KDE and counterfactual sampling S C
Revised: 3.2.2 Density estimation S KDE and counterfactual sampling S C

Original: The dimensionality D of hyper-representations z in [33], as well as in our work, is relativelyhigh due to the challenge of compressing weights w .
Revised: The dimensionality D of hyper-representations z in [37], as well as in our work, is relatively high due to the challenge of compressing weights w .

Original: Fitting a probability density model tosuch a high-dimensional distribution is feasible by making a conditional independence assumption: p ( z ( j ) | z ( k ) , w ) =
Revised: Fitting a probability density model to such a high-dimensional distribution is feasible by making a conditional independence assumption: p ( z ( j ) | z ( k ) , w ) =

Original: p ( z ( j ) | w ) , where z ( j ) is the j -th dimensionality of the embedding z .
Revised: p ( z ( j ) | w ) , where z ( j ) is the j -th dimensionality of the embedding z .

Original: Tomodel the distribution of each j -th dimensionality, we choose kernel density estimation (KDE), asit is a powerful yet simple, non-parametric and deterministic method with a single hyperparameter.
Revised: To model the distribution of each j -th dimensionality, we choose kernel density estimation (KDE), as it is a powerful yet simple, non-parametric and deterministic method with a single hyperparameter.

Original: We ﬁt a KDE to the M anchor samples { z ( j ) i } Mi =1 of each dimension j , and draw samples z ( j ) fromthat distribution:
Revised: We fit a KDE to the M anchor samples { z ( j ) i } Mi =1 of each dimension j , and draw samples z ( j ) from that distribution:

Original: z ( j ) ⇠ p ( z ( j ) )
Revised: z ( j ) ∼ p ( z ( j ) )

Original: =
Revised: =

Original: 1 Mh [Equation]
Revised: 1 Mh (cid:80) Mi

Original:  K [Equation]
Revised: =1 K ( z ( j ) − z

Original:  z ( j ) i h ) , where K ( x ) = (2 ⇡ )  1 / 2 exp (  x 2 2 ) isthe Gaussian kernel and h is a bandwidth hyperparameter.
Revised: ( j ) i h ) , where K ( x ) = (2 π ) − 1 / 2 exp ( − x 2 2 ) is the Gaussian kernel and h is a bandwidth hyperparameter.

Original: The samples of each dimension z ( j ) areconcatenated to form samples z ⇤ =
Revised: The samples of each dimension z ( j ) are concatenated to form samples z ∗ =

Original: [ z (1) , z (2) , · · · , z ( D ) ] .
Revised: [ z (1) , z (2) , · · · , z ( D ) ] .

Original: This method is denoted as S KDE .
Revised: This method is denoted as S KDE .

Original: As a sanity check, we invert the S KDE method and explicitly draw samples from regions not populatedby anchor samples, i.e. with low probability according to the KDE.
Revised: As a sanity check, we invert the S KDE method and explicitly draw samples from regions not populated by anchor samples, i.e. with low probability according to the KDE.

Original: This method, denoted as S C ,essentially samples counterfactual embeddings and similarly to S U is expected to perform poorly.
Revised: This method, denoted as S C , essentially samples counterfactual embeddings and similarly to S U is expected to perform poorly.

Original: 3.2.3 Neighbor sampling S Neigh
Revised: 3.2.3 Neighbor sampling S Neigh

Original: This method models the neighborhood topology to reduce the dimensionality and facilitate sampling. We approximate the neighborhood topology ofthe anchor samples  z i using an invertible functionk : R D!
Revised: Sampling neighbors of anchor samples { z i } could be a simple and effective sampling strategy, but due to high sparsity of the hyper-representation space this strategy results in poor-quality samples. We therefore propose to use a neighborhood-based dimensionality reduction function k : R D

Original:  R d with n i = k
Revised: → R d that maps

Original: ( z i ) .
Revised: z i to low-dimensional embeddings n i ∈ R d where sampling is facilitated.

Original: We choose d = 3 as a sufﬁciently low dimensional space tosample from. Samples n ⇤ are uniformly drawn from the box surrounding the d -dimensional reductionn
Revised: The assumption is that due to the low dimensionality of R d (we choose d = 3 ) there will be fewer low-probability regions, so that uniform sampling in R d can be effective. Specifically, given low-dimensional embeddings n i = k

Original: ⇤ ⇠ U ( min ( n ) , max ( n )) and mapped back to hyper-representations z ⇤ =
Revised: ( z i ) , we sample n ∗ uniformly from the cube: n ∗ ∼ U ( min ( n ) , max ( n )) . Samples n ∗ are then mapped back to hyper-representations z ∗ =

Original: k  1 ( n ⇤ ) .
Revised: k − 1 ( n ∗ ) .

Original: For k , we usea neighborhood-based dimensionality reduction method UMAP [26].
Revised: To preserve the neighborhood topology of R D in R d and enable mapping back to R D , we choose k to be an approximate inverse neighborhood-based dimensionality reduction function based on UMAP [28].

Original: 3.2.4 Latent space GAN S 
Revised: 3.2.4 Latent space GAN S GAN

Original: GAN

Original: A common choice for generative representation learning is generative adversarial networks(GANs)
Revised: A common choice for generative representation learning is generative adversarial networks (GANs) [12].

Original: While training a GAN directly to generate weights is a promising yet challenging avenue for future research [31], we found the GAN framework to work reasonably well when trained onthe hyper-representations.
Revised: While training a GAN directly to generate weights is a promising yet challenging avenue for future research [35], we found the GAN framework to work reasonably well when trained on the hyper-representations.

Original: This idea follows [24, 12] that showed improved training stability and efﬁciency compared to training GANs on inputs directly.
Revised: This idea follows [26, 13] that showed improved training stability and efficiency compared to training GANs on inputs directly.

Original: We train a generator G : R d ! R D with z ⇤ =
Revised: We train a generator G : R d → R D with z ∗ =

Original: G ( n ⇤ ) to generate samples in hyper-representation space from the Gaussian noise n ⇤ .
Revised: G ( n ∗ ) to generate samples in hyper-representation space from the Gaussian noise n ∗ .

Original: We choose d =16 as a compromise between size and capacity.
Revised: We choose d = 16 as a compromise between size and capacity.

Original: See a detailed architecture of our GAN in Appendix E.
Revised: See a detailed architecture of our GAN in Appendix E.

Original:   4.1 Experimental Setup
Revised:  Experiments 4.1 Experimental Setup

Original: We train and evaluate our approaches on four image classiﬁcation datasets: MNIST [21], SVHN [29],
Revised: We train and evaluate our approaches on four image classification datasets: MNIST [24], SVHN [32],

Original: CIFAR-10 [20] and STL-10 [4].
Revised: CIFAR-10 [23], STL-10 [5].

Original: For each dataset, there is a model zoo that we use to train anautoencoder following [33].
Revised: For each dataset, there is a model zoo that we use to train an autoencoder following [37].

Revised: Model zoos: In practice, there are already many available model zoos, e.g., on Hugging Face or

Revised: GitHub, that can be used for hyper-representation learning and sampling.

Revised: Unfortunately, these zoos are not systematically constructed and require further effort to mine and evaluate.

Revised: Therefore, in order to control the experiment design, ensure feasibility and reproducibility, we generate novel or use the model zoos of [37, 38] created in a systematic way.

Revised: With controlled experiments, we aim to develop and evaluate inductive biases and methods to train and utilize hyper-representation, which can be scaled up efficiently to large-scale and non-systematically constructed zoos later.

Original: Model zoos: In each image dataset, a zoo contains M = 1000 convolutional networks of the same architecture with three convolutional layers and two fully-connected layers.
Revised: For each image dataset, a zoo contains M = 1000 convolutional networks of the same architecture with three convolutional layers and two fully-connected layers.

Original: Varying only in the randomseeds, all models of the zoo are trained for 50 epochs with the same hyperparameters following [33].
Revised: Varying only in the random seeds, all models of the zoo are trained for 50 epochs with the same hyperparameters following [37].

Original: To integrate higher diversity in the zoo, initial weights are uniformly sampled from a wider range ofvalues rather than using well-tuned initializations of [10, 14].
Revised: To integrate higher diversity in the zoo, initial weights are uniformly sampled from a wider range of values rather than using well-tuned initializations of [11, 15].

Original: Each zoo is split in the train (70%), validation (15%) and test (15%) splits.
Revised: Each zoo is split in the train (70%), validation (15%) and test (15%) splits.

Original: To incorporate the learning dynamics, we train autoencoders onthe models trained for 21-25 epochs following [33].
Revised: To incorporate the learning dynamics, we train autoencoders on the models trained for 21-25 epochs following [37].

Original: Here the models have already achieved high performance, but have not fully converged.
Revised: Here the models have already achieved high performance, but have not fully converged.

Original: The development in the remaining epochs of each model aretreated as hold-out data to compare against.
Revised: The development in the remaining epochs of each model are treated as hold-out data to compare against.

Original: We use the MNIST and SVHN zoos from [33] and basedon them create the CIFAR-10 and STL-10 zoos.
Revised: We use the MNIST and SVHN zoos from [37] and based on them create the CIFAR-10 and STL-10 zoos.

Original: Details on the zoos can be found in Appendix A.
Revised: Details on the zoos can be found in Appendix A.

Original: Training details: We train separate hyper-representations on each of the model zoos.
Revised: Experimental details: We train separate hyper-representations on each of the model zoos.

Original: Images andlabels are not used to train the hyper-representations (see § 2).
Revised: Images and labels are not used to train the hyper-representations (see § 2).

Original: Using the proposed sampling methods(§ 3.2), we generate new embeddings and decode them to weights.
Revised: Using the proposed sampling methods (§ 3.2), we generate new embeddings and decode them to weights.

Original: We evaluate sampled populationsas initializations (epoch 0) and by ﬁne-tuning for up to 25 epochs.
Revised: We evaluate sampled populations as initializations (epoch 0) and by fine-tuning for up to 25 epochs.

Original: We distinguish between in-datasetand transfer-learning.
Revised: We distinguish between in-dataset and transfer-learning.

Original: For in-dataset, the same image dataset is used for training and evaluating ourhyper-representations and baselines.
Revised: For in-dataset, the same image dataset is used for training and evaluating our hyper-representations and baselines.

Original: For transfer-learning, hyper-representations (and pre-trainedmodels in baselines) are trained on a source dataset. All populations are evaluated and ﬁne-tunedon a different target dataset.
Revised: For transfer-learning, hyper-representations (and pre-trained models in baselines) are trained on a source dataset, then all populations are evaluated and fine-tuned on a different target dataset.

Original: Full details on training, including infrastructure and compute is detailedin the Appendix B.
Revised: Full details on training, including infrastructure and compute is detailed in the Appendix B.

Original: Baselines: As the ﬁrst baseline, we consider the autoencoder of [33], which is same as ours butwithout the proposed layer-wise loss-normalization (LWLN, § 3.1).
Revised: Baselines: As the first baseline, we consider the autoencoder of [37], which is same as ours but without the proposed layer-wise loss-normalization (LWLN, § 3.1).

Original: We combine this autoencoderwith the S KDE30 sampling method and, hence, denote it as B KDE 30 .
Revised: We combine this autoencoder with the S KDE30 sampling method and, hence, denote it as B KDE 30 .

Original: We consider two other baselinesbased on training models with stochastic gradient descent (SGD): training from scratch on the targetclassiﬁcation task B T , and training on a source followed by ﬁne-tuning on the target task B F .
Revised: We consider two other baselines based on training models with stochastic gradient descent (SGD): training from scratch on the target classification task B T , and training on a source followed by fine-tuning on the target task B F .

Original: Thelatter remains one of the strongest transfer learning baselines [3, 8, 19].
Revised: The latter remains one of the strongest transfer learning baselines [4, 9, 22].

Original: Reproducibility, reliability and comparability: We compare populations of at least 50 models toevaluate each method reliably.
Revised: Reproducibility, reliability and comparability: We compare populations of at least 50 models to evaluate each method reliably.

Original: We report standard deviation in Tables 1-2 and statistical signiﬁcance,effect size and 95% conﬁdence interval in Appendix F. To ensure fairness and comparability, allmethods share training hyperparameters.
Revised: We report standard deviation in Tables 1-2 and statistical significance, effect size and 95% confidence interval in Appendix F. To ensure fairness and comparability, all methods share training hyperparameters.

Original: Fine-tuning uses the hyperparameters of the target domain.
Revised: Fine-tuning uses the hyperparameters of the target domain.

Original: 4.2 Results
Revised: 4.2 Results

Original: In the following, we ﬁrst analyze the learned hyper-representations further justifying our samplingmethods and assumptions made in § 3.2.
Revised: In the following, we first analyze the learned hyper-representations further justifying our sampling methods and assumptions made in § 3.2.

Original: We then conﬁrm the effectiveness of our approach for modelinitialization without and with ﬁne-tuning in the in-dataset and transfer learning settings.
Revised: We then confirm the effectiveness of our approach for model initialization without and with fine-tuning in the in-dataset and transfer learning settings.

Original: 4.2.1 Hyper-Representations are Robust and Smooth
Revised: 4.2.1 Hyper-Representations are Robust and Smooth

Original: We evaluate the robustness and smoothness of the hyper-representation space with two experiments.
Revised: We evaluate the robustness and smoothness of the hyper-representation space with two experiments on the SVHN zoo.

Original: First, to evaluate robustness, we add different levels of noise to the embeddings of the test set to create˜ z , decode them to model weights ˜ w and compute models’ accuracies on the classiﬁcation task.
Revised: First, to evaluate robustness, we add different levels of noise to the embeddings of the test set to create ˜ z , decode them to model weights ˜ w and compute models’ accuracies on the SVHN classification task.

Original: Wefound that  hyper-representations are robust to noise as large levels of relative noise >10% are requiredto affect performance (Figure 3, c).
Revised: We found that both the baseline as well as our hyper-representations are robust to noise as large levels of relative noise >10% are required to affect performance (Figure 3, a,c).

Original: Second, to probe for smoothness, we linearly interpolate betweenthe test set embeddings (i) along  the trajectory of the same model at different epochs ( z [-i,ep 
-]
Revised: Second, to probe for smoothness, we linearly interpolate between the test set embeddings (i) along [Figure] the trajectory of the same model at different epochs ( z [-i,ep 
-]ep and z

Original: andz i,ep 25 ) and (ii) between 250 random pairs of embeddings on the trajectories of different models ( z iand z j ).
Revised: ,ep 25 ) and (ii) between 250 random pairs of embeddings on the trajectories of different models ( z i and z j ).

Original: We decode the interpolated embeddings and compute models’ accuracies on the classiﬁcationtask.
Revised: We decode the interpolated embeddings and compute models’ accuracies on the classification task.

Original: We found remarkably smooth development of accuracy along the interpolation in both schemes(Figure 3, d).
Revised: For our model, we found remarkably smooth development of accuracy along the interpolation in both schemes (Figure 3, d).

Original: The lack of ﬂuctuations along and between trajectories support both local and globalnotions of smoothness in hyper-representation space.
Revised: The lack of fluctuations along and between trajectories support both local and global notions of smoothness in hyper-representation space.

Original: [Figure] For the baseline autoencoder (without LWLN) decoded models all perform close to 10% accuracy, sothese representations do not support similar notions of smoothness (Figure 3, b), while robustness canbe misleading, since the accuracy even without adding noise is already low (Figure 3, a).
Revised: For the baseline autoencoder (without LWLN) decoded models all perform close to 10% accuracy, so these representations do not support similar notions of smoothness (Figure 3, b), while robustness can be misleading, since the accuracy even without adding noise is already low (Figure 3, a).

Original: Therefore,
Revised: Therefore,

Original: LWLN together with regularizations added to the autoencoder allow for learning robust and smoothhyper-representation.
Revised: LWLN together with regularizations added to the autoencoder allow for learning robust and smooth hyper-representation.

Original: This property makes sampling from that representation more meaningful aswe show next.
Revised: This property makes sampling from that representation more meaningful as we show next.

Original: 4.2.2 Sampling for In-dataset Initialization
Revised: 4.2.2 Sampling for In-dataset Initialization

Original: [Figure] Comparison between samplingmethods: We evaluate the performanceof different sampled populations (obtained with LWLN) without ﬁne-tuninggenerated weights.
Revised: Comparison between sampling methods: We evaluate the performance of different sampled populations (obtained with LWLN) without fine-tuning generated weights.

Original: On MNIST, allsampled models except those obtainedusing S U and S C perform better thanrandom initialization (10% accuracy),but worse than models trained fromscratch B T for 50 epochs (Figure 4).
Revised: On MNIST, all sampled models except those obtained using S U and S C perform better than random initialization (10% accuracy), but worse than models trained from scratch B T for 25 epochs (Figure 4).

Original: Distribution-based samples ( S KDE and S GAN ) perform betterthan neighborhood based samples ( S Neigh ).
Revised: Distribution-based samples ( S KDE and S GAN ) perform better than neighborhood based samples ( S Neigh ).

Original: The populations based on the top 30% perform better thantheir 100% counterparts with S KDE30 as the strongest sampling method overall.
Revised: The populations based on the top 30% perform better than their 100% counterparts with S KDE30 as the strongest sampling method overall.

Original: This demonstratesthat the learned hyper-representation and sampling methods are able to capture complex subtletiesin weight space differentiating high and low performing models.[Figure]
Revised: This demonstrates that the learned hyper-representation and sampling methods are able to capture complex subtleties in weight space differentiating high and low performing models.

Original: [Table] Comparison to the baseline hyperrepresentations: We also compare S KDE30that is based on our autoencoder with layerwise loss normalization (LWLN) to thebaseline autoencoder using the same sampling method ( B KDE 30 ) without ﬁne-tuning.
Revised: Comparison to the baseline hyper-representations: We also compare S KDE30 that is based on our autoencoder with layer-wise loss normalization (LWLN) to the baseline autoencoder using the same sampling method ( B KDE 30 ) without fine-tuning.

Original: On all datasets except for MNIST, S KDE30considerably outperform B KDE 30 with thelatter performing just above 10% (randomguessing), see Table 1 (rows with epoch0).
Revised: On all datasets except for MNIST, S KDE30 considerably outperform B KDE 30 with the latter performing just above 10% (random guessing), see Table(rows with epoch 0).

Original: We attribute the success of LWLN totwo main factors.
Revised: We attribute the success of LWLN to two main factors.

Original: First, LWLN preventsthe collapse of reconstruction to the mean(compare Figure 2-top to Figure 2-bottom).
Revised: First, LWLN prevents the collapse of reconstruction to the mean (compare Figure 2 top to bottom).

Original: Second, by ﬁxing the weak links, thereconstructed models perform signiﬁcantlybetter.
Revised: Second, by fixing the weak links, the reconstructed models perform significantly better (see Appendix C for more results).

Original: We also evaluated input-to-outputfeature normalization, s.t. encoder and decoder operate on normalized weights, but empirically foundit did not work as well as a normalization just for the loss.

Original: More details are provided in Appendix C.

Original: In-dataset ﬁne-tuning: When ﬁne-tuning, our S KDE 30 and baseline B KDE 30 appear to graduallyconverge to similar performance (Table 1).
Revised: In-dataset fine-tuning: When fine-tuning, our S KDE 30 and baseline B KDE 30 appear to gradually converge to similar performance (Table 1).

Original: While unfortunate, this result aligns well with previous ﬁndings that longer training and enough data make initialization less important [28, 15, 30].
Revised: While unfortunate, this result aligns well with previous findings that longer training and enough data make initialization less important [30, 17, 34]. [Table]

Original: We alsocompare S KDE 30 and B KDE 30 to training models from scratch ( B T ).
Revised: We also compare S KDE 30 and B KDE 30 to training models from scratch ( B T ).

Original: On all four datasets, both oursand the baseline hyper-representations outperform B T when generated weights are ﬁne-tuned forthe same number of epochs as B T .
Revised: On all four datasets, both ours and the baseline hyper-representations outperform B T when generated weights are fine-tuned for the same number of epochs as B T .

Original: Notably, on MNIST and SVHN generated weights ﬁne-tuned for25 epochs are even better than B T run for 50 epochs.
Revised: Notably, on MNIST and SVHN generated weights fine-tuned for 25 epochs are even better than B T run for 50 epochs.

Original: Comparison to 50 epochs is more fair though,since the hyper-representations were trained on model weights trained for up to 25 epochs.
Revised: Comparison to 50 epochs is more fair though, since the hyper-representations were trained on model weights trained for up to 25 epochs.

Original: Theseﬁndings show that the models initialized with generated weights learn faster achieving better resultsin 25 epochs than B T in 50 epochs.
Revised: These findings show that the models initialized with generated weights learn faster achieving better results in 25 epochs than B T in 50 epochs.

Original: Sampling ensembles: We found that a  useful by-product of learninghyper-representations is the ability to generate high-performantensembles at almost no extra computational cost, since bothsampling and generation are computationally cheap.
Revised: Sampling ensembles: We found that a potentially useful byproduct of learning hyper-representations is the ability to generate high-performant ensembles at almost no extra computational cost, since both sampling and generation are computationally cheap.

Original: To demonstratethis effect, we compare ensembles formed using the baselineautoencoder ( B KDE 30 ) and ours ( S KDE 30 ) to the ensembles composedof models trained from scratch for 25 epochs ( B T ).
Revised: To demonstrate this effect, we compare ensembles formed using the baseline autoencoder ( B KDE 30 ) and ours ( S KDE 30 ) to the ensembles composed of models trained from scratch for 25 epochs ( B T ) on

Original: On SVHN,ensembles generated using the baseline B KDE 30 stagnate below20% (Figure 5).
Revised: SVHN. Ensembles generated using the baseline B KDE 30 stagnate below 20% (Figure 5).

Original: In contrast, ensembles generated using ourS KDE 30gracefully improve with the ensemble size outperforming single
Revised: In contrast, ensembles generated using our

Original:  B T models and almost matching B T ensembles on both datasetswith enough models in the ensembles.
Revised: S KDE 30 gracefully improve with the ensemble size outperforming single B T models and almost matching B T ensembles with enough models in the ensembles.

Original: Remarkably, on SVHN, theaverage test accuracy of generated ensembles of 15 models is 77.6%,which is considerably higher than 70.7% of models trained on SVHN for 50 epochs.
Revised: Remarkably, the average test accuracy of generated ensembles of 15 models is 77.6%, which is considerably higher than 70.7% of models trained on SVHN for 50 epochs.

Original: We conclude, hyper-representations learnedwith LWLN generate models which are not only performant, but also diverse.
Revised: We conclude that hyper-representations learned with LWLN generate models that are not only performant, but also diverse.

Original: Although generatingensembles requires learning hyper-representation and model zoo ﬁrst, we assume that in future sucha hyper-representation can be trained once and reused in unseen scenarios as we tentatively explorebelow (see results in Table 3 and the discussion therein).
Revised: Although generating ensembles requires learning hyper-representation and model zoo first, we assume that in future such a hyper-representation can be trained once and reused in unseen scenarios as we tentatively explore below (see results in Table 3 and the discussion therein).

Original: Do reconstructed models becomesimilar to the original duringﬁne-tuning?
Revised: Do reconstructed models become similar to the original during fine-tuning?

Original: Sampled hyperrepresentations often learn fasterand to a higher performance thanthe population of models they weretrained on (Table 1).
Revised: Sampled hyperrepresentations often learn faster and to a higher performance than the population of models they were trained on (Table 1).

Original: We thereforeexplore the question, if reconstructedmodels develop in weight space inthe same direction as their original,or ﬁnd a different solution.
Revised: We therefore explore the question, if reconstructed models develop in weight space in the same direction as their original, or find a different solution.

Original: We foundthat the reconstructed models afterone epoch of ﬁne-tuning perform similarly well as their originals  and slightly outperform from thereon (Figure 6, left).
Revised: On SVHN, we found that the reconstructed models ( ˆ w ) after one epoch of fine-tuning perform similar to their originals ( w ) and slightly outperform from there on (Figure 6, left).

Original: At the same time, pairs of original and reconstructed models move further apartand become less aligned in weight space (Figure 6, right).
Revised: At the same time, pairs of original and reconstructed models move further apart and become less aligned in weight space (Figure 6, right).

Original: It appears that reconstructed modelsperform better and explore different solutions in weight space to do so.
Revised: It appears that reconstructed models perform better and explore different solutions in weight space to do so.

Original: This conﬁrms the intuition,that hyper-representations impress useful structure on decoded weights.
Revised: This confirms the intuition that hyper-representations impress useful structure on decoded weights.

Original: A pass through encoder anddecoder thus results not just in a noisy reconstruction of the original sample.
Revised: A pass through encoder and decoder thus results not just in a noisy reconstruction of the original sample.

Original: Instead, it maps to adifferent region on the loss surface, which leads to faster learning and better solutions.
Revised: Instead, it maps to a different region on the loss surface, which leads to faster learning and better solutions.

Original: Consideringthe ensembling results above (Figure 5), hyper-representations do not collapse on a single goodsolution, but decode to diverse, useful weights.
Revised: Combining this with the ensembling results in Figure 5, hyper-representations do not collapse to a single solution, but decode to diverse and useful weights.

Original: 4.2.3 Sampling  for Cross-dataset Initialization
Revised: 4.2.3 Sampling Initializations for Transfer Learning

Original: Setup: We investigate the effectiveness of our method in a transfer-learning setup across imagedatasets.
Revised: Setup: We investigate the effectiveness of our method in a transfer-learning setup across image datasets.

Original: In particular, we report transfer learning results from SVHN to MNIST and from STL-10to CIFAR-10 as two representative scenarios.
Revised: In particular, we report transfer learning results from SVHN to MNIST and from STLto CIFAR-10 as two representative scenarios.

Original: Results on the other pairs of datasets can be foundin Appendix F.
Revised: Results on the other pairs of datasets can be found in Appendix F.

Original: In these experiments, pre-trained models B F and the hyper-representation modelare trained on a source domain.
Revised: In these experiments, pre-trained models B F and the hyper-representation model are trained on a source domain.

Original: Subsequently, the pre-trained models B F and the samples S KDE ,
Revised: Subsequently, the pre-trained models B F and the samples S KDE ,

Original: S Neigh and S GAN are ﬁne-tuned on the target domain.
Revised: S Neigh and S GAN are fine-tuned on the target domain.

Original: The baseline approach ( B T ) is based on trainingmodels from scratch 
Revised: The baseline approach ( B T ) is based on training models from scratch on the target domain.

Original: [Table]
Revised: [Table]

Original: Results: When transfer learning is performed from SVHN to MNIST, the sampled populationson average learn faster and achieve signiﬁcantly higher performance than the B T baseline andgenerally compares favorably to B F (Figure 7, Table 2).
Revised: Results: When transfer learning is performed from SVHN to MNIST, the sampled populations on average learn faster and achieve significantly higher performance than the B T baseline and generally compares favorably to B F (Figure 7, Table 2).

Original: In the STL-10 to CIFAR-10 experiment, allpopulations appear to saturate with only small differences in their performances (Table 2).
Revised: In the STL-10 to CIFAR-10 experiment, all populations appear to saturate with only small differences in their performances (Table 2).

Original: Differentsampling methods perform differently at the beginning versus the end of transfer learning.
Revised: Different sampling methods perform differently at the beginning versus the end of transfer learning.

Original: Generally,
Revised: Generally,

Original: S KDE30 performs better in the ﬁrst epochs, while all methods perform comparably at the end oftransfer-learning.
Revised: S KDE30 performs better in the first epochs, while all methods perform comparably at the end of transfer-learning.

Original: These discrepancies underline the difﬁculty of developing a single strong samplingmethod, which is an interesting area of future research.
Revised: These discrepancies underline the difficulty of developing a single strong sampling method, which is an interesting area of future research.

Original: We further found that all datasets are usefulsources for all targets (see Appendix F).
Revised: We further found that all datasets are useful sources for all targets (see Appendix F).

Original: Interestingly and other than in related work [27], eventransfer from the simpler to harder datasets (e.g., MNIST to SVHN) improves performance.
Revised: Interestingly and other than in related work [29], even transfer from the simpler to harder datasets (e.g., MNIST to SVHN) improves performance.

Original: Thismight be explained by the ability of hyper-representations to capture a generic inductive prior usefulacross different domains, which we further investigate next.
Revised: This might be explained by the ability of hyper-representations to capture a generic inductive prior useful across different domains, which we further investigate next.

Original: [Table]
Revised: [Table]

Original: Conditioning on unseen zoos: We explore if[Table] the hyper-representation trained on the modelsof one zoo (e.g. MNIST) can reconstruct theweights of another unseen zoo (e.g. SVHN).
Revised: Conditioning on unseen zoos: We explore if the hyper-representation trained on the models of one zoo (e.g. MNIST) can reconstruct the weights of another unseen zoo (e.g. SVHN).

Original: This can be useful to enable generation ofweights for novel tasks without the need toretrain a hyper-representation.
Revised: This can be useful to enable generation of weights for novel tasks without the need to retrain a hyper-representation.

Original: This is analogous to instance-conditioned GANs that recentlywere able to generate images from unseen domains without retraining GANs [2].
Revised: This is analogous to instance-conditioned GANs that recently were able to generate images from unseen domains without retraining GANs [2].

Original: Our resultsin Table 3 show that while the performance on the unseen zoos is reduced, it is still well aboverandom guessing (10%), especially when multiple model weights are sampled and ensembled. 
Revised: Our results in Table 3 show that while the performance on the unseen zoos is reduced, it is still well above random guessing (10%), especially when multiple model weights are sampled and ensembled. This is promising, as the hyper-representations were trained on single-dataset zoos.

Revised: 4.2.4 Sampling Initializations for Unseen Architectures

Original: Thisis notable, as the hyper-representations were trained on single-dataset zoos.
Revised: Generalization to unseen large architectures with complex connectivity (ResNet, MobileNet, and EfficientNet) is a very interesting and ambitious research problem. As a step towards that goal, we perform experiments in which we attempted to use our hyper-representation beyond the same simple architecture. Surprisingly, our results indicate the promise of leveraging the hyper-representation for more diverse architectures and settings.

Original: 4.3 Limitations of Zoos with Small Models
Revised: Further experiments investigating the cross-architecture generalization capabilities of hyper-representations can be found in Appendix D. Setup: With this experiment, we aim to verify if it is possible to adapt our approach to architectures not seen during training, e.g., with skip connections and/or with more layers. We follow the transferlearning setup of § 4.2.3 and use an existing MNIST hyper-representation to sample weights as initializiation for training on SVHN. However, we now also vary the architecture.

Revised: While the decoder outputs a fixed-sized vector of weights, we can assign these weights to new architectures by either making sure that the new architecture still has the same number of parameters or by initializing randomly the extra parameters introduced.

Revised: Specifically, we create three cases: (1) we add ResNet-style skip connections [16] (1x1 conv) to the convolutional layers (3-conv + res-skip), (2) re-distribute the weights to smaller four convolutional layers (4-conv), (3) re-distribute to smaller four convolutional layers and add identity skip connections (4-conv + id skip

Revised: [Table]

Revised: Results: Surprisingly, despite training our hyper-representation on the models of the same architecture, generated weights for all three cases outperform random initialization and converge significantly faster across all the variations (Table 4).

Revised: In all the variations even just after 5 epochs the models with generated weights are better than training the baseline for 50 epochs.

Revised: In the 3-conv + res-skip experiments, some models in both populations did not learn, which leads to high standard deviation.

Revised: Further analysis is required to explain the gains of our approach in this challenging setup.

Revised: To extend and scale up our method further, future work could combined it with the methods of growing networks [3, 42], so that some layers are generated while some are initialized in a sophisticated way to preserve the functional form of the network.

Revised: 4.3 Limitations of Zoos with Small Models

Original: To thoroughly investigate different methods and make experiments feasible, we chose to use the modelzoos of the same  scale as in [33].
Revised: To thoroughly investigate different methods and make experiments feasible, we chose to use the model zoos of the same small scale as in [37].

Original: While on MNIST and SVHN, the architectures of such model zoosallowed us to achievea reasonably high performance, on CIFAR-10 and STL-10, the performance ofall populations is limited by the low capacity of the models  architecture.
Revised: While on MNIST and SVHN, the architectures of such model zoos allowed us to achieve high performance, on CIFAR-10 and STL-10, the performance of all populations is limited by the low capacity of the models zoo’s architecture.

Original: The models saturate at around50% and 40% accuracy, respectively.
Revised: The models saturate at around 50% and 40% accuracy, respectively.

Original: The sampled populations reach the saturation point and ﬂuctuate,but cannot outperform the baselines, see Appendix F.
Revised: The sampled populations reach the saturation point and fluctuate, but cannot outperform the baselines, see Appendix F for details.

Original: We hypothesize that due to the high remainingloss, the weight updates are correspondingly large without converging or improving performance.
Revised: We hypothesize that due to the high remaining loss, the weight updates are correspondingly large without converging or improving performance.

Original: This may cause the weights to contain relatively little signal and high noise.
Revised: This may cause the weights to contain relatively little signal and high noise.

Original: Indeed, learning hyperrepresentations on the CIFAR-10 and STL-10 zoos was difﬁcult and never reached similar reconstruction performance as in the MNIST and SVHN zoos. That in turn additionally limits the performanceof the sampled populations. Future work will therefore focus on architectures with higher capacity.
Revised: Larger model architectures might mitigate this behaviour. Corresponding model zoos have recently been made available in [38] to tackle this issue 1 .

Original:  Related Work
Revised:  Related Work

Original: HyperNetworks: Representation learning on neural networks is typically based on HyperNetworksthat learn low-dimensional structure of model weights to generate weights in a deterministic fashion [13, 1, 18, 39].
Revised: HyperNetworks: Recently, representation learning on neural networks is typically based on HyperNetworks that learn low-dimensional structure of model weights to generate weights in a deterministic fashion [14, 1, 21, 45].

Original: HyperNetworks have also been extended to meta-learning by conditioning weightgeneration on data [40, 32].
Revised: HyperNetworks have also been extended to meta-learning by conditioning weight generation on data [46, 36].

Original: Closely related to our work, HyperGANs [31] can sample model weightsby combining the hypernetworks and the GAN framework.
Revised: Closely related to our work, HyperGANs [35] can sample model weights by combining the hypernetworks and the GAN framework.

Original: Similarly, [7] allow for sampling modelweights by conditioning the hypernetwork on a noise vector.
Revised: Similarly, [8] allow for sampling model weights by conditioning the hypernetwork on a noise vector.

Original: However, training hypernetwork-basedmethods require input data (e.g. images) to feed to the neural networks.
Revised: However, training hypernetwork-based methods require input data (e.g. images) to feed to the neural networks.

Original: In practice, there may alreadybe large collections of trained models, while their training data may not always be accessible.
Revised: In practice, there may already be large collections of trained models, while their training data may not always be accessible.

Original: Learning representations of model weights without data, called hyper-representations, has been recentlyintroduced in [33].
Revised: Learning representations of model weights without data, called hyper-representations, has been recently introduced in [37].

Original: Our methods build on that work to allow for better reconstruction and sampling.[6] showed that network weights of a single model can be accurately reconstructed, however in ourwork we leverage the autoencoder to train a representation of the entire model zoo.
Revised: Our methods build on that work to allow for better reconstruction and sampling. [7] showed that given a few parameters of a network, the remaining values of a single model can be accurately reconstructed. However, in our work we leverage the autoencoder to train a representation of the entire model zoo.

Original:  Transfer Learning: Transfer learning via ﬁne-tuning aims at re-using models and their learnedknowledge from a source to a target task [38, 3, 8, 27, 19].
Revised: Very recently, [33] use diffusion on a population of models to generate model weights for the original task via prompting. Transfer Learning: Transfer learning via fine-tuning aims at re-using models and their learned knowledge from a source to a target task [44, 4, 9, 29, 22].

Original: Transfer learning models makes trainingless expensive, boosts performance, or allows to train on datasets with very few samples and hasbeen applied on a wide range of domains [42].
Revised: Transfer learning models makes training less expensive, boosts performance, or allows to train on datasets with very few samples and has been applied on a wide range of domains [48].

Original: The common transfer learning methods however onlyconsider transferring from a single model, and so disregard the large variety of pre-trained modelsand potential beneﬁt of combining them.
Revised: The common transfer learning methods however only consider transferring from a single model, and so disregard the large variety of pre-trained models and potential benefit of combining them.

Original: Knowledge distillation: Our work is related to [37, 22, 34] that allow to distill knowledge from amodel zoo into a single network.
Revised: Knowledge distillation: Our work is related to [43, 25, 39] that allow to distill knowledge from a model zoo into a single network.

Original: Knowledge distillation overcomes the inherent limitation of transferlearning by transferring the knowledge from many large teacher models to a relatively small studentmodel [23, 35].
Revised: Knowledge distillation overcomes the inherent limitation of transfer learning by transferring the knowledge from many large teacher models to a relatively small student model [25, 39].

Original: Knowledge distillation however requires the source models at training as in [23]and at inference as in[35] thus increasing memory cost.
Revised: Knowledge distillation however requires the source models at training as in [25] and at inference as in[39] thus increasing memory cost.

Original: Further, the learned knowledge cannot beshared between different target models.
Revised: Further, the learned knowledge cannot be shared between different target models.

Original: Learnable initialization [5, 41] provide methods to improveinitialization by leveraging the meta-learning and gradient-ﬂow ideas.
Revised: Learnable initialization [6, 47] provide methods to improve initialization by leveraging the meta-learning and gradient-flow ideas.

Original: In contrast to knowledgedistillation and learnable initialization, we train a hyper-representation of a model zoo in a latentspace, which is a more general and powerful approach that can enable sampling an ensemble, propertyestimation, improved initialization and implicit knowledge distillation across datasets.
Revised: In contrast to knowledge distillation and learnable initialization, we train a hyper-representation of a model zoo in a latent space, which is a more general and powerful approach that can enable sampling an ensemble, property estimation, improved initialization and implicit knowledge distillation across datasets.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose a new method to sample from hyper-representations to generate neuralnetwork weights in one forward pass.
Revised: In this paper, we propose a new method to sample from hyper-representations to generate neural network weights in one forward pass.

Original: We extend the training objective of hyper-representationsby a novel layer-wise loss normalization which is key to the capability of generating functionalmodels.
Revised: We extend the training objective of hyper-representations by a novel layer-wise loss normalization which is key to the capability of generating functional models.

Original: Our method allows us to generate diverse populations of model weights, which show highperformance as ensembles.
Revised: Our method allows us to generate diverse populations of model weights, which show high performance as ensembles.

Original: We evaluate sampled models both in-dataset as well as in transfer learningand ﬁnd them capable to outperform both models trained from scratch, as well as pre-trained andﬁne-tuned models.
Revised: We evaluate sampled models both in-dataset as well as in transfer learning and find them capable to outperform both models trained from scratch, as well as pre-trained and finetuned models.

Original: Populations of sampled models generally learn faster and achieve statisticallysigniﬁcantly higher performance.
Revised: Populations of sampled models, even for some unseen architectures, generally learn faster and achieve statistically significantly higher performance.

Original: This demonstrates that such hyper-representation can be used asa generative model for neural network weights and therefore might serve as a building block fortransfer learning from different domains, meta learning or continual learning. 
Revised: This demonstrates that such hyperrepresentation can be used as a generative model for neural network weights and therefore might serve as a building block for transfer learning from different domains, meta learning or continual learning. Acknowledgments This work was partially funded by Google Research Scholar Award, the University of

Revised: St.Gallen Basic Research Fund, and project PID2020-117142GB-I00 funded by MCIN/ AEI /10.13039/501100011033.

Revised: We are thankful to Michael Mommert for editorial support.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section.•
Revised: [Yes] See Section.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data are proprietary.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...[Table] (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s [Table]
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: contributions and scope?

Original: [Yes][Table] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.3 and 6(c)
Revised: [Yes] See Section 4.3 and(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [No] The method[Table] we propose is a general machine learning method.
Revised: [No] The method we propose is a general machine learning method.

Original: It comes with all the risks involved,but in our opinion does not pose any particular risk.
Revised: It comes with all the risks involved, but in our opinion does not pose any particular risk.

Original: [Table] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A] 3.

Original: If you ran experiments...[Table] (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?[Yes] Code,
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original:  hyper-representations of selected model zoos are included in the supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they 
Revised: [Yes] Code, hyper-representations of selected model zoos are included in the supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: were chosen)? [Yes] All the details are speciﬁed in the Appendix A and B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] All the details are specified in the Appendix A and B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We report means and std for all experiments in the main paper and provide additional details about median, CI95 on the median, p-value andeffect size in the Appendix F (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] We report means and std for all experiments in the main paper and provide additional details about median, CI95 on the median, p-value and effect size in the Appendix F (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix B 
Revised: [Yes] See Appendix B 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] See Section A (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] See Section A (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] We have trained new model zoos and will make them publicly available via an URL 
Revised: [Yes] We have trained new model zoos and will make them publicly available via an URL (d) Did you discuss whether

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] Our experiments use publicly available image datasets, whichwe cite.
Revised: [N/A] Our experiments use publicly available image datasets, which we cite.

Original: The model zoos and code for hyper-representation learning was publishedon github and zenodo by the authors of the original paper under Creative Commons Attribution 4.0.
Revised: The model zoos and code for hyper-representation learning was published on github and zenodo by the authors of the original paper under Creative Commons Attribution 4.0.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] Our experiments rely on common computer vision datasets and do not contain personally identiﬁable information nor offensive 
Revised: [N/A] Our experiments rely on common computer vision datasets and do not contain personally identifiable information nor offensive content.

Original: content. If you used crowdsourcing or conducted research with human subjects... 
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [Table]
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: A Model Zoo Details

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable? [N/A]
Revised: The model zoos are generated following the method of [37, 38] An overview of the model zoos is given in in Table 5. All model zoos share one general CNN architecture, outlined in Table 6.

Original:  [Table]
Revised: The hyperparameter choices for each of the population are listed in Table 7. The hyperparameters are chosen to generate zoos with smooth, continuous development and spread in performance. [Table] [Table]

Revised: Hyper-representations are learned with an autoencoder based on multi-head self-attention.

Original: (c) Did you include the estimated hourly wage paid to participants and  the total amount
Revised: The architecture is outlined in Figure 8. Convolutional and fully connected neurons are embedded to token embeddings of dimension d token . Learned position encodings are added to provide relational information. A learned compression token (CLS) is appended to the sequence of token embeddings.

Revised: The sequence of token embeddings is passed to N layers layers of multi-head self-attention with

Revised: N heads heads with hidden embedding dimension d hidden .

Revised: The CLS token is compressed to the bottleneck of dimension d z with an MLP or a linear layer.

Revised: For the decoder, an MLP or a linear layer maps the bottleneck to a sequence of token embeddings.

Revised: The sequence is passed through another stack of multi-head self-attention, which is symmetric to the encoder.

Revised: Debedders map the token embeddings back to convolutional and fully connected neurons.

Revised: The reconstruction and contrastive loss are balanced with a parameter β .

Revised: The contrastive loss is computed on the embeddings z mapped through a projection head ¯z =

Revised: p ( z , where p is a learned MLP with four layers with 400 neurons each and ¯z has 50 dimensions.

Revised: In Table 8, the exact hyper-parameters for each of the hyper-representation are listed to reproduce our results.

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: C Evaluation of Layer-Wise Loss Normalization

Revised: To evaluate layer-wise loss normalization, we compare two hyper-representations with comparable reconstruction.

Revised: Both have a R 2 = 1 − mse (ˆ w , w ) mse ( w mean , w as a measure of the explained variance of around 70%.

Revised: One is trained trained with the baseline hyper-representation MSE, the other with layer-wise-normalization.

Revised: Figures 9 and 10 show the distribution of weights per layer before and after reconstruction, as well as the accuracy distribution of populations on the SVHN image test set.

Revised: With the basline learning scheme in Figure 9, the distributions in layers 3 and 4 do not match.

Revised: In these layers, the original weight distribution is smaller, and so there is only a small error even if the reconstructions predicts the mean.

Revised: These layers become a weak link of the reconstructed models, and cause performance around random guessing.

Revised: With layer-wise loss normalization in Figure 10, the weight distribution between the layers becomes more similar.

Revised: As a consequence, the reconstruction error is more evenly distributed across the layers, there are no single layers that aren’t reconstructed at all.

Revised: This appears to allow information to flow forward through the model, and significantly improves the performance of reconstructed models.

Revised: We find layer-wise-normalization necessary to reconstruct or sample functional models across all populations, where the weights are unevenly distributed.

Revised: D Hyper-Representation Analysis

Revised: In this section, we detail the analysis of hyper-representations.

Revised: We begin with their geometry, followed by the distributions of individual dimensions of hyper-representations, and finally investigate robustness and smoothness.

Revised: Embeddings in Hyper-Representation Space Populate a Hyper-Sphere

Revised: We analyse the geometry of hyper-representations z .

Revised: The space of hyper-representations is bounded to a high dimensional box by a tanh activation.

Revised: Surprisingly, hyper-representations do not populate the entire space, but sections on a shell of a high-dimensional sphere.

Revised: Figure 11 shows the distribution of the norm of the embeddings of the MNIST zoo.

Revised: All embeddings are distributed on a small band between lengthand 12, therefore they must populate the shell of a hyper-sphere.

Revised: In Figure 12 we investigate pairwise cosine distances between the embeddings of the MNIST zoo.

Revised: The majority of the embeddings populate the region between 0.6 and 0.8.

Revised: The outliers around 1.0 are embeddings of the same model at different epochs.

Revised: This indicates that models are not entirely orthogonal, but mutually equally far apart, populating a section of the shell of the hyper-sphere.

Revised: While hyper-spheres are commonly found in embeddings of contrastive learning [19], in our experiments hyper-spheres form even without a contrastive loss.

Revised: Properties of the models embedded on that hyper-sphere can be predicted from hyper-representations, therefore the topology on the sphere appears to encode model properties.

Revised: Distributions of Dimensions of Embeddings in Hyper-Representation Encode Properties Previous work showed that linear probing from hyper-representations accurately predicts i.e. model accuracy.

Revised: In these linear probes, the individual z dimensions each linearly contribute to accuracy predictions.

Revised: This allows us investigate z dimensions independently.

Revised: Figure 13 shows examples for the distribution of selected individual dimensions of hyper-representations z .

Revised: On the left are the distribution of the entire population, on the right of the top 30 % performing models.

Revised: The individual dimensions show different types of distributions, with different modes.

Revised: Most have a zero mean and span 3/4 of the available range, but some collapse to either − 1 or 1 .

Revised: Further, the distributions also differ in at least some dimension between the entire population, and the better performing split of the population.

Revised: Generalization Capabilities of Hyper-Representations to Diverse Model Zoos There are certain architectural changes such as adding/removing/changing pooling layers and nonlinearity that do not change the number of parameters (the dimensionality of the input/output required by our approach).

Revised: These changes as well as changes of hyperparameters used to train models in a zoo may drastically alter the distribution of weights and pose a challenge to the proposed approach.

Revised: Modern neural networks (ResNet, MobileNet, EfficientNet, etc.) are often trained with very different hyperparameters.

Revised: With the experiment below, we investigate the generalization capabilities of hyper-representations to suchchanges, which might be important for modern large-scale settings as well.

Revised: Setup: We experimentally evaluate generalizability of the proposed approach on models trained with a different choice of nonlinearity or other hyperparameters with two experiments (a and b).

Revised: To that end, in addition to the original SVHN test zoo (zoo 1), we use two more diverse SVHN zoos (zoo 2 and zoo 3).

Revised: In zoo 2, in addition to random seed, models differ in the activation (tanh, relu, gelu, sigmoid), l2-regularization (0, 0.001, 0.1) and dropout (0,0.3,0.5).

Revised: In zoo 3 (extending zoo 2), we increase the diversity further by additionally varying the initialization method (uniform, normal, kaiming-uniform, kaiming-normal) and the learning rate (0.0001, 0.001, 0.01).

Revised: Experiment (a): We first evaluate our original encoder-decoder trained on a model zoo varying in random seed only.

Revised: For evaluation, we pass the test splits of zoo 2 and zoo 3 through the encoderdecoder.

Revised: We measure the reconstruction R 2 score of the original encoder-decoder on the diverse test zoos.

Revised: Results: Our results (Table D) indicate that our original encoder-decoder can still encode and decode weights even in such a challenging setting, although there is an expected drop of performance.

Revised: Experiment (a): We next evaluate if hyper-representations can be trained on diverse zoos.

Revised: For this experiment, we train a hyper-representation on the train split of zoo 3.

Revised: With this, we aim to show that training hyper-representations on diverse zoos improves generalization capabilities further.

Revised: Results: Our results show that training on diverse zoos is a much more difficult task to optimize, hence the reconstruction on the original zoo degrades.

Revised: It nonetheless improves the reconstruction results on the test split of the diverse zoos 2 and 3.

Revised: This indicates that varying seeds and hyperparameters may be different aspects of complexity that need to be considered.

Original: [Table]
Revised: [Table]

Revised: E Sampling Methods E.1 VAE

Revised: A common extension of the autoencoder of [37] to enable sampling from its latent representation is to make the autoencoder variational [20].

Revised: In our experiments, VAEs could not be trained to satisfactory reconstruct model weights without unweighting the KL-divergence to insignificance essentially making it deterministic as in [37].

Original: spent on participant compensation?
Revised: Empirically, embeddings in hyper-representations are mapped on the shell of a sphere (see Section D) and leave the inside of the sphere entirely empty. On the other hand, a gaussian prior allocates most of the probability mass near the center of the sphere. It therefore appears plausible that the two may be incompatible. That issue of non-compatible priors is well known.

Revised: [10] find that regularizing embeddings and decoder yields equally smooth representation spaces as VAEs without restrictions to specific priors.

Revised: During training of hyper-representations, both encoder and decoder are regularized with a small ℓ 2 penalty.

Revised: Further, dropout is applied throughout the autoencoder, which servers as another regularizer and adds blurryness to the embeddings.

Revised: The combination of dropout, the erasing augmentation and the contrastive loss further regularizes the hyper-representation space.

Original: [N/A]
Revised: In all our sampling methods, we draw samples from probability distributions, which effectively disconnects the drawn samples from training embeddings. E.2 Latent Space GAN Details The generator and discriminator of our GAN consist of four fully-connected layers interleaved with ReLU nonlinearities.

Revised: The same architecture and training hyperparameters are used for all experiments.

Revised: The generator’s input is a Gaussian noise n ∗ of dimensionality d = 16 , the hidden dimensionalities are 128, 256 and 512, and the output dimensionality is equal to the hyper-representation length D .

Revised: The discriminator’s input is D -dimensional, the hidden dimensionalities are 1024, 512 and 256, and the output dimensionality is a scalar denoting either a real or fake sample.

Revised: The discriminator is regularized with Spectral Norm [31].

Revised: The discriminator and generator are trained for 1000 epochs and batch size 32 using Adam with a two time-scale update rule [

Revised: F Full Experiment Results F.1 Digit Domain

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]