Original: Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights
Revised: Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights

Original: Abstract
Revised: Abstract

Original: Learning representations of neural network weights given a model zoo is an emerging and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation.
Revised: Learning representations of neural network weights given a model zoo is an emerging and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation.

Original: Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation , which captures intrinsic and extrinsic properties of the models in the zoo.
Revised: Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation , which captures intrinsic and extrinsic properties of the models in the zoo.

Original: In this work, we extend hyper-representations for generative use to sample new model weights.
Revised: In this work, we extend hyper-representations for generative use to sample new model weights.

Original: We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and several sampling methods based on the topology of hyper-representations.
Revised: We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and several sampling methods based on the topology of hyper-representations.

Original: The models generated using our methods are diverse, performant and capable to outperform strong baselines as evaluated on several downstream tasks: initialization, ensemble sampling and transfer learning.
Revised: The models generated using our methods are diverse, performant and capable to outperform strong baselines as evaluated on several downstream tasks: initialization, ensemble sampling and transfer learning.

Original: Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.
Revised: Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.

Original:  Introduction
Revised:  Introduction

Original: Over the last decade, countless neural network models have been trained and uploaded to different model hubs.
Revised: Over the last decade, countless neural network models have been trained and uploaded to different model hubs.

Original: Many factors such as random initialization and no global optimum ensure that the trained models are different from one another.
Revised: Many factors such as random initialization and no global optimum ensure that the trained models are different from one another.

Original: What could we learn from such a population of neural network models?
Revised: What could we learn from such a population of neural network models?

Original: Since the parameter space of neural networks is complex and high-dimensional, representation learning from such populations (often referred to as model zoos) has become an emerging and challenging area.
Revised: Since the parameter space of neural networks is complex and high-dimensional, representation learning from such populations (often referred to as model zoos) has become an emerging and challenging area.

Original: Recent work along that direction has demonstrated the ability of such learned representations to capture intrinsic and extrinsic properties of the models in a zoo [40, 37, 27].
Revised: Recent work along that direction has demonstrated the ability of such learned representations to capture intrinsic and extrinsic properties of the models in a zoo [40, 37, 27].

Original: According to [37], NNs populate a low dimensional manifold, which can be learned with an autoencoder via self-supervised learning directly from the model paramters (weights and biases) without access to the original image data and labels.
Revised: According to [37], NNs populate a low dimensional manifold, which can be learned with an autoencoder via self-supervised learning directly from the model paramters (weights and biases) without access to the original image data and labels.

Original: This so called hyper-representation has been demonstrated to be useful to predict several model properties such as accuracy, hyperparameters or architecture configurations.
Revised: This so called hyper-representation has been demonstrated to be useful to predict several model properties such as accuracy, hyperparameters or architecture conﬁgurations.

Original: However, [37] focused on discriminative downstream tasks by exploiting the encoder only.
Revised: However, [37] focused on discriminative downstream tasks by exploiting the encoder only.

Original: We take one step further and extend their work towards the generative downstream tasks by sampling model weights directly from the task-agnostic hyper-representation.
Revised: We take one step further and extend their work towards the generative downstream tasks by sampling model weights directly from the task-agnostic hyper-representation.

Original: To that end, we introduce a layer-wise normalization that improves the quality of decoded neural network weights significantly.
Revised: To that end, we introduce a layer-wise normalization that improves the quality of decoded neural network weights signiﬁcantly.

Original: Based on a careful analysis of the geometry, smoothness and robustness of this space, we also propose several sampling methods to generate weights in a single forward pass from the hyper-representation.
Revised: Based on a careful analysis of the geometry, smoothness and robustness of this space, we also propose several sampling methods to generate weights in a single forward pass from the hyper-representation.

Original: We evaluate our approach on four image datasets and three generative downstream tasks of (i) model initialization, (ii) ensemble sampling, and (iii) transfer learning.
Revised: We evaluate our approach on four image datasets and three generative downstream tasks of (i) model initialization, (ii) ensemble sampling, and (iii) transfer learning.

Original: Our results demonstrate its capability to out-perform previous hyper-representation learning and conventional baselines.
Revised: Our results demonstrate its capability to out-perform previous hyper-representation learning and conventional baselines.

Original: Previous work on generating model weights proposed (Graph) HyperNetworks [14, 45, 21], Bayesian
Revised: Previous work on generating model weights proposed (Graph) HyperNetworks [14, 45, 21], Bayesian

Original: HyperNetworks [8], HyperGANs [35] and HyperTransformers [46] for neural architecture search, model compression, ensembling, transfer- or meta-learning.
Revised: HyperNetworks [8], HyperGANs [35] and HyperTransformers [46] for neural architecture search, model compression, ensembling, transfer- or meta-learning.

Original: These methods learn representations by using images and labels of the target domain.
Revised: These methods learn representations by using images and labels of the target domain.

Original: In contrast, our approach only uses model weights and does not need access to underlying data samples and labels – an emergent use case, e.g. of deep learning monitoring services or model hubs.
Revised: In contrast, our approach only uses model weights and does not need access to underlying data samples and labels – an emergent use case, e.g. of deep learning monitoring services or model hubs.

Original: In addition to the ability to generate novel and diverse model weights, compared to previous works our approach (a) can generate novel weights conditionally on model zoos from unseen tasks and (b) can be conditioned on the latent factors of the underlying hyperrepresentation.
Revised: In addition to the ability to generate novel and diverse model weights, compared to previous works our approach (a) can generate novel weights conditionally on model zoos from unseen tasks and (b) can be conditioned on the latent factors of the underlying hyperrepresentation.

Original: Notably, both (a) and (b) can be done without the need to retrain hyper-representations.
Revised: Notably, both (a) and (b) can be done without the need to retrain hyper-representations.

Original: The results suggest our approach (Figure 1) to be a promising step towards a general purpose hyper-representation encapsulating knowledge of model zoos to advance different downstream tasks.
Revised: The results suggest our approach (Figure 1) to be a promising step towards a general purpose hyper-representation encapsulating knowledge of model zoos to advance different downstream tasks.

Original: The hyper-representations and code to reproduce our results are available at https://github.com/HSG-AIML/NeurIPS_2022-Generative_Hyper_Representations .
Revised: The hyper-representations and code to reproduce our results are available at https://github.com/HSG-AIML/NeurIPS_2022-Generative_Hyper_Representations .

Original:  Background: Training Hyper-Representations
Revised:  Background: Training Hyper-Representations

Original: We summarize the first stage of our method that corresponds to learning a hyper-representation of a population of neural networks, called a model zoo [37].
Revised: We summarize the ﬁrst stage of our method that corresponds to learning a hyper-representation of a population of neural networks, called a model zoo [37].

Original: In [37] and this paper, a model zoo consists of models trained on the same task such as CIFAR-10 image classification [23].
Revised: In [37] and this paper, a model zoo consists of models trained on the same task such as CIFAR-10 image classiﬁcation [23].

Original: Specifically, a hyper-representation is learned using an autoencoder ˆ w i
Revised: Speciﬁcally, a hyper-representation is learned using an autoencoder ˆ w i

Original: = h ( g ( w i )) on a zoo of M models { w i } M 1 , where w i is the flattened vector of dimension N of all the weights of the i -th model.
Revised: = h ( g ( w i )) on a zoo of M models { w i } M 1 , where w i is the ﬂattened vector of dimension N of all the weights of the i -th model.

Original: The encoder g compresses vector w i to fixed-size hyper-representation z
Revised: The encoder g compresses vector w i to ﬁxed-size hyper-representation z

Original: i = g ( w i ) of lower dimension.
Revised: i = g ( w i ) of lower dimension.

Original: The decoder h decompresses the hyper-representation to the reconstructed vector
Revised: The decoder h decompresses the hyper-representation to the reconstructed vector

Original: ˆ w i .
Revised: ˆ w i .

Original: Both encoder and decoder are built on a self-attention block [41].
Revised: Both encoder and decoder are built on a self-attention block [41].

Original: The samples from model zoos are understood as sequences of convolutional or fully connected neurons.
Revised: The samples from model zoos are understood as sequences of convolutional or fully connected neurons.

Original: Each of the neurons is encoded as a token embedding and concatenated to form a sequence.
Revised: Each of the neurons is encoded as a token embedding and concatenated to form a sequence.

Original: The sequence is passed through several layers of multi-head self-attention.
Revised: The sequence is passed through several layers of multi-head self-attention.

Original: Afterwards, a special compression token summarizing the entire sequence is linearly compressed to the bottleneck.
Revised: Afterwards, a special compression token summarizing the entire sequence is linearly compressed to the bottleneck.

Original: The output is fed through a tanh-activation to achieve a bounded latent space
Revised: The output is fed through a tanh-activation to achieve a bounded latent space

Original: z i for the hyper-representation.
Revised: z i for the hyper-representation.

Original: The decoder is symmetric to the encoder, the embeddings are linearly decompressed from hyper-representations z i and position encodings are added.
Revised: The decoder is symmetric to the encoder, the embeddings are linearly decompressed from hyper-representations z i and position encodings are added.

Original: Training is done in a multi-objective fashion, minimizing the composite loss L = β L MSE +(1 − β )
Revised: Training is done in a multi-objective fashion, minimizing the composite loss L =  L MSE +(1   )

Original: L c , where L c is a contrastive loss and L MSE is a weight reconstruction loss (see details in [37]).
Revised: L c , where L c is a contrastive loss and L MSE is a weight reconstruction loss (see details in [37]).

Original: We can write the latter in a layer-wise way to facilitate our discussion in § 3.1:
Revised: We can write the latter in a layer-wise way to facilitate our discussion in § 3.1:

Original: [Equation]
Revised: [Equation]

Original: where ˆ w ( l ) i , w ( l ) i are reconstructed and original weights for the l -th layer of the i -th model in the zoo.
Revised: where ˆ w ( l ) i , w ( l ) i are reconstructed and original weights for the l -th layer of the i -th model in the zoo.

Original: The contrastive loss L c leverages two types of data augmentation at train time to impose structure on the latent space: permutation exploiting inherent symmetries of the weight space and random erasing.
Revised: The contrastive loss L c leverages two types of data augmentation at train time to impose structure on the latent space: permutation exploiting inherent symmetries of the weight space and random erasing.

Original:  Methods
Revised:  Methods

Original: In the following, we present (i) layer-wise loss normalization to ensure that decoded models are performant, and (ii) sampling methods to generate diverse populations of models.
Revised: In the following, we present (i) layer-wise loss normalization to ensure that decoded models are performant, and (ii) sampling methods to generate diverse populations of models.

Original: 3.1 Layer-Wise Loss Normalization
Revised: 3.1 Layer-Wise Loss Normalization

Original: We observed that hyper-representations as proposed by [37] decode to dysfunctional models, with performance around random guessing.
Revised: We observed that hyper-representations as proposed by [37] decode to dysfunctional models, with performance around random guessing.

Original: To alleviate that, we propose a novel layer-wise loss normalization (LWLN), which we motivate and detail in the following.
Revised: To alleviate that, we propose a novel layer-wise loss normalization (LWLN), which we motivate and detail in the following.

Original: Due to the MSE training loss in (1), the reconstruction error can generally be expected to be uniformly distributed over all weights and layers of the weight vector w .
Revised: Due to the MSE training loss in (1), the reconstruction error can generally be expected to be uniformly distributed over all weights and layers of the weight vector w .

Original: However, the weight magnitudes of many of our zoos are unevenly distributed across different layers.
Revised: However, the weight magnitudes of many of our zoos are unevenly distributed across different layers.

Original: In these zoos, the even distribution of reconstruction errors lead to undesired effects.
Revised: In these zoos, the even distribution of reconstruction errors lead to undesired effects.

Original: Layers with broader distributions and large-magnitude weights are reconstructed well, while layers with narrow distributions and small-magnitude weights are disregarded.
Revised: Layers with broader distributions and large-magnitude weights are reconstructed well, while layers with narrow distributions and small-magnitude weights are disregarded.

Original: The latter layers can become a weak link in the reconstructed models, causing performance to drop significantly down to random guessing.
Revised: The latter layers can become a weak link in the reconstructed models, causing performance to drop signiﬁcantly down to random guessing.

Original: The top row of Figure 2 shows an example of a baseline hyper-representation learned on the zoo of SVHN models [32].
Revised: The top row of Figure 2 shows an example of a baseline hyper-representation learned on the zoo of SVHN models [32].

Original: Common initialization schemes [15, 11] produce distributions with different scaling factors per layer, so the issue is not an artifact of the zoos, but can exist in real world model populations.
Revised: Common initialization schemes [15, 11] produce distributions with different scaling factors per layer, so the issue is not an artifact of the zoos, but can exist in real world model populations.

Original: Similarly, recent work on generating models normalizes weights to boost performance [21].
Revised: Similarly, recent work on generating models normalizes weights to boost performance [21].

Original: In order to achieve equally accurate reconstruction across the layers, we introduce a layer-wise loss normalization (LWLN) with the mean µ l and standard deviation σ l of all weights in layer l estimated over the train split of the zoo:L ¯ MSE = MN M (cid:88) i =
Revised: In order to achieve equally accurate reconstruction across the layers, we introduce a layer-wise loss normalization (LWLN) with the mean µ l and standard deviation  l of all weights in layer l estimated over the train split of the zoo:

Original: L (cid:88) l = (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)

Original: ˆ w ( l ) i − µ l σ l − w ( l ) i − µ l σ l (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)  = MN M (cid:88) i = L (cid:88) l = ∥ ˆ w ( l ) i − w ( l ) i ∥σ 2 l .
Revised: [Equation]

Original: 3.2 Sampling from Hyper-Representations
Revised: 3.2 Sampling from Hyper-Representations

Original: We introduce methods to draw diverse and high-quality samples z ∗ ∼ p ( z ) from the learned hyperrepresentation space to generate model weights w ∗ =
Revised: We introduce methods to draw diverse and high-quality samples z ⇤ ⇠ p ( z ) from the learned hyperrepresentation space to generate model weights w ⇤ =

Original: h ( z ∗ ) .
Revised: h ( z ⇤ ) .

Original: Such sampling is facilitated if there is knowledge on the topology of the space spanned by z .
Revised: Such sampling is facilitated if there is knowledge on the topology of the space spanned by z .

Original: One way to achieve that is to train a variational autoencoder (VAE) with a predefined prior [20] instead of the autoencoder of [37].
Revised: One way to achieve that is to train a variational autoencoder (VAE) with a predeﬁned prior [20] instead of the autoencoder of [37].

Original: While training VAEs on common domains such as images has become well-understood and feasible, in our relatively novel weight domain, we found it problematic (see details in Appendix E).
Revised: While training VAEs on common domains such as images has become well-understood and feasible, in our relatively novel weight domain, we found it problematic (see details in Appendix E).

Original: Other generative methods avoid a predefined prior of VAEs, either by analyzing the topology of the space learned by the autoencoder or fitting a separate density estimation model on top of the learned representation [26, 13].
Revised: Other generative methods avoid a predeﬁned prior of VAEs, either by analyzing the topology of the space learned by the autoencoder or ﬁtting a separate density estimation model on top of the learned representation [26, 13].

Original: These methods assume the representation space to have strong regularities.
Revised: These methods assume the representation space to have strong regularities.

Original: The hyper-representation space learned by the autoencoder of [37] is already regularized by dropout regularization applied to the encoder and decoder as in [10].
Revised: The hyper-representation space learned by the autoencoder of [37] is already regularized by dropout regularization applied to the encoder and decoder as in [10].

Original: The contrastive loss component requiring similar models to be embedded close to each other may also improve the regularity of the representation space.
Revised: The contrastive loss component requiring similar models to be embedded close to each other may also improve the regularity of the representation space.

Original: Empirically, we found our layer-wise loss normalization (LWLN) to further regularize the representation space by ensuring robustness and smoothness (see Figure 3 in § 4).
Revised: Empirically, we found our layer-wise loss normalization (LWLN) to further regularize the representation space by ensuring robustness and smoothness (see Figure 3 in § 4).

Original: Given the smoothness and robustness of the learned hyper-representation space, we follow [26, 13, 10] in estimating the density and topology to draw samples from a regularized autoencoder.
Revised: Given the smoothness and robustness of the learned hyper-representation space, we follow [26, 13, 10] in estimating the density and topology to draw samples from a regularized autoencoder.

Original: To that end, we introduce three strategies to sample from that space: S KDE , S Neigh , S GAN .
Revised: To that end, we introduce three strategies to sample from that space: S KDE , S Neigh , S GAN .

Original: To model the density and topology in representation space, we use the embeddings of the train set as anchor samples { z i } .
Revised: To model the density and topology in representation space, we use the embeddings of the train set as anchor samples { z i } .

Original: We observe that many anchor samples from { z i } correspond to the models with relatively poor accuracy (Figure 2), so to improve the quality of sampled weights, we consider the variants of these methods using only those embeddings of training samples corresponding to the top 30% performing models.
Revised: We observe that many anchor samples from { z i } correspond to the models with relatively poor accuracy (Figure 2), so to improve the quality of sampled weights, we consider the variants of these methods using only those embeddings of training samples corresponding to the top 30% performing models.

Original: We denote these sampling methods as S KDE30 , S Neigh30 , S GAN30 respectively.
Revised: We denote these sampling methods as S KDE30 , S Neigh30 , S GAN30 respectively.

Original: These methods can potentially decrease sample diversity, however, we found that the generated weights are still diverse enough (e.g. to construct high-performant ensembles, Figure 5).
Revised: These methods can potentially decrease sample diversity, however, we found that the generated weights are still diverse enough (e.g. to construct high-performant ensembles, Figure 5).

Original: Finally, as baseline and sanity check we explore sampling uniformly in representation space S U and sampling in low-probability regions S C .
Revised: Finally, as baseline and sanity check we explore sampling uniformly in representation space S U and sampling in low-probability regions S C .

Original: 3.2.1 Uniform S U
Revised: 3.2.1 Uniform S U

Original: As a naive baseline, we draw samples uniformly in hyper-representation space (bounded by tanh, § 2) and denote it as S U .
Revised: As a naive baseline, we draw samples uniformly in hyper-representation space (bounded by tanh, § 2) and denote it as S U .

Original: This is naive, because we found that the embeddings z populate only sections of a shell of a high-dimensional sphere (see Figures 11 and 12 in Appendix D).
Revised: This is naive, because we found that the embeddings z populate only sections of a shell of a high-dimensional sphere (see Figures 11 and 12 in Appendix D).

Original: So most of the uniform samples lie in the low-probability regions of the space and are not expected to be decoded to useful models.
Revised: So most of the uniform samples lie in the low-probability regions of the space and are not expected to be decoded to useful models.

Original: 3.2.2 Density estimation S KDE and counterfactual sampling S C
Revised: 3.2.2 Density estimation S KDE and counterfactual sampling S C

Original: The dimensionality D of hyper-representations z in [37], as well as in our work, is relatively high due to the challenge of compressing weights w .
Revised: The dimensionality D of hyper-representations z in [37], as well as in our work, is relatively high due to the challenge of compressing weights w .

Original: Fitting a probability density model to such a high-dimensional distribution is feasible by making a conditional independence assumption: p ( z ( j ) | z ( k ) , w ) =
Revised: Fitting a probability density model to such a high-dimensional distribution is feasible by making a conditional independence assumption: p ( z ( j ) | z ( k ) , w ) =

Original: p ( z ( j ) | w ) , where z ( j ) is the j -th dimensionality of the embedding z .
Revised: p ( z ( j ) | w ) , where z ( j ) is the j -th dimensionality of the embedding z .

Original: To model the distribution of each j -th dimensionality, we choose kernel density estimation (KDE), as it is a powerful yet simple, non-parametric and deterministic method with a single hyperparameter.
Revised: To model the distribution of each j -th dimensionality, we choose kernel density estimation (KDE), as it is a powerful yet simple, non-parametric and deterministic method with a single hyperparameter.

Original: We fit a KDE to the M anchor samples { z ( j ) i } Mi =1 of each dimension j , and draw samples z ( j ) from that distribution: z ( j ) ∼ p ( z ( j ) )
Revised: We ﬁt a KDE to the M anchor samples { z ( j ) i } Mi =1 of each dimension j , and draw samples z ( j ) from that distribution: z ( j ) ⇠ p ( z ( j ) )

Original: =
Revised: =

Original: 1 Mh (cid:80) Mi
Revised: 1 Mh [Equation]

Original: =1 K ( z ( j ) − z ( j ) i h ) , where K ( x ) = (2 π ) − 1 / 2 exp ( − x 2 2 ) is the Gaussian kernel and h is a bandwidth hyperparameter.
Revised: ) , where K ( x ) = (2 ⇡ )  1 / 2 exp (  x 2 2 ) is the Gaussian kernel and h is a bandwidth hyperparameter.

Original: The samples of each dimension z ( j ) are concatenated to form samples z ∗ =
Revised: The samples of each dimension z ( j ) are concatenated to form samples z ⇤ =

Original: [ z (1) , z (2) , · · · , z ( D ) ] .
Revised: [ z (1) , z (2) , · · · , z ( D ) ] .

Original: This method is denoted as S KDE .
Revised: This method is denoted as S KDE .

Original: As a sanity check, we invert the S KDE method and explicitly draw samples from regions not populated by anchor samples, i.e. with low probability according to the KDE.
Revised: As a sanity check, we invert the S KDE method and explicitly draw samples from regions not populated by anchor samples, i.e. with low probability according to the KDE.

Original: This method, denoted as S C , essentially samples counterfactual embeddings and similarly to S U is expected to perform poorly.
Revised: This method, denoted as S C , essentially samples counterfactual embeddings and similarly to S U is expected to perform poorly.

Original: 3.2.3 Neighbor sampling S Neigh
Revised: 3.2.3 Neighbor sampling S Neigh

Original: Sampling neighbors of anchor samples { z i } could be a simple and effective sampling strategy, but due to high sparsity of the hyper-representation space this strategy results in poor-quality samples.
Revised: Sampling neighbors of anchor samples { z i } could be a simple and effective sampling strategy, but due to high sparsity of the hyper-representation space this strategy results in poor-quality samples.

Original: We therefore propose to use a neighborhood-based dimensionality reduction function k : R D 
Revised: We therefore propose to use a neighborhood-based dimensionality reduction function k : R D !

Original: → R d that maps
Revised: R d that maps

Original: z i to low-dimensional embeddings n i ∈ R d where sampling is facilitated.
Revised: z i to low-dimensional embeddings n i 2 R d where sampling is facilitated.

Original: The assumption is that due to the low dimensionality of R d (we choose d = 3 ) there will be fewer low-probability regions, so that uniform sampling in R d can be effective.
Revised: The assumption is that due to the low dimensionality of R d (we choose d = 3 ) there will be fewer low-probability regions, so that uniform sampling in R d can be effective.

Original: Specifically, given low-dimensional embeddings n i = k
Revised: Speciﬁcally, given low-dimensional embeddings n i = k

Original: ( z i ) , we sample n ∗ uniformly from the cube: n ∗ ∼ U ( min ( n ) , max ( n )) .
Revised: ( z i ) , we sample n ⇤ uniformly from the cube: n ⇤ ⇠ U ( min ( n ) , max ( n )) .

Original: Samples n ∗ are then mapped back to hyper-representations z ∗ =
Revised: Samples n ⇤ are then mapped back to hyper-representations z ⇤ =

Original: k − 1 ( n ∗ ) .
Revised: k  1 ( n ⇤ ) .

Original: To preserve the neighborhood topology of R D in R d and enable mapping back to R D , we choose k to be an approximate inverse neighborhood-based dimensionality reduction function based on UMAP [28].
Revised: To preserve the neighborhood topology of R D in R d and enable mapping back to R D , we choose k to be an approximate inverse neighborhood-based dimensionality reduction function based on UMAP [28].

Original: 3.2.4 Latent space GAN S GAN
Revised: 3.2.4 Latent space GAN S GAN

Original: A common choice for generative representation learning is generative adversarial networks (GANs) [12].
Revised: A common choice for generative representation learning is generative adversarial networks (GANs) [12].

Original: While training a GAN directly to generate weights is a promising yet challenging avenue for future research [35], we found the GAN framework to work reasonably well when trained on the hyper-representations.
Revised: While training a GAN directly to generate weights is a promising yet challenging avenue for future research [35], we found the GAN framework to work reasonably well when trained on the hyper-representations.

Original: This idea follows [26, 13] that showed improved training stability and efficiency compared to training GANs on inputs directly.
Revised: This idea follows [26, 13] that showed improved training stability and efﬁciency compared to training GANs on inputs directly.

Original: We train a generator G : R d → R D with z ∗ =
Revised: We train a generator G : R d ! R D with z ⇤ =

Original: G ( n ∗ ) to generate samples in hyper-representation space from the Gaussian noise n ∗ .
Revised: G ( n ⇤ ) to generate samples in hyper-representation space from the Gaussian noise n ⇤ .

Original: We choose d = 16 as a compromise between size and capacity.
Revised: We choose d = 16 as a compromise between size and capacity.

Original: See a detailed architecture of our GAN in Appendix E.
Revised: See a detailed architecture of our GAN in Appendix E.

Original:  Experiments 4.1 Experimental Setup
Revised:  Experiments 4.1 Experimental Setup

Original: We train and evaluate our approaches on four image classification datasets: MNIST [24], SVHN [32],
Revised: We train and evaluate our approaches on four image classiﬁcation datasets: MNIST [24], SVHN [32],

Original: CIFAR-10 [23], STL-10 [5].
Revised: CIFAR-10 [23], STL-10 [5].

Original: For each dataset, there is a model zoo that we use to train an autoencoder following [37].
Revised: For each dataset, there is a model zoo that we use to train an autoencoder following [37].

Original: Model zoos: In practice, there are already many available model zoos, e.g., on Hugging Face or
Revised: Model zoos: In practice, there are already many available model zoos, e.g., on Hugging Face or

Original: GitHub, that can be used for hyper-representation learning and sampling.
Revised: GitHub, that can be used for hyper-representation learning and sampling.

Original: Unfortunately, these zoos are not systematically constructed and require further effort to mine and evaluate.
Revised: Unfortunately, these zoos are not systematically constructed and require further effort to mine and evaluate.

Original: Therefore, in order to control the experiment design, ensure feasibility and reproducibility, we generate novel or use the model zoos of [37, 38] created in a systematic way.
Revised: Therefore, in order to control the experiment design, ensure feasibility and reproducibility, we generate novel or use the model zoos of [37, 38] created in a systematic way.

Original: With controlled experiments, we aim to develop and evaluate inductive biases and methods to train and utilize hyper-representation, which can be scaled up efficiently to large-scale and non-systematically constructed zoos later.
Revised: With controlled experiments, we aim to develop and evaluate inductive biases and methods to train and utilize hyper-representation, which can be scaled up efﬁciently to large-scale and non-systematically constructed zoos later.

Original: For each image dataset, a zoo contains M = 1000 convolutional networks of the same architecture with three convolutional layers and two fully-connected layers.
Revised: For each image dataset, a zoo contains M = 1000 convolutional networks of the same architecture with three convolutional layers and two fully-connected layers.

Original: Varying only in the random seeds, all models of the zoo are trained for 50 epochs with the same hyperparameters following [37].
Revised: Varying only in the random seeds, all models of the zoo are trained for 50 epochs with the same hyperparameters following [37].

Original: To integrate higher diversity in the zoo, initial weights are uniformly sampled from a wider range of values rather than using well-tuned initializations of [11, 15].
Revised: To integrate higher diversity in the zoo, initial weights are uniformly sampled from a wider range of values rather than using well-tuned initializations of [11, 15].

Original: Each zoo is split in the train (70%), validation (15%) and test (15%) splits.
Revised: Each zoo is split in the train (70%), validation (15%) and test (15%) splits.

Original: To incorporate the learning dynamics, we train autoencoders on the models trained for 21-25 epochs following [37].
Revised: To incorporate the learning dynamics, we train autoencoders on the models trained for 21-25 epochs following [37].

Original: Here the models have already achieved high performance, but have not fully converged.
Revised: Here the models have already achieved high performance, but have not fully converged.

Original: The development in the remaining epochs of each model are treated as hold-out data to compare against.
Revised: The development in the remaining epochs of each model are treated as hold-out data to compare against.

Original: We use the MNIST and SVHN zoos from [37] and based on them create the CIFAR-10 and STL-10 zoos.
Revised: We use the MNIST and SVHN zoos from [37] and based on them create the CIFAR-10 and STL-10 zoos.

Original: Details on the zoos can be found in Appendix A.
Revised: Details on the zoos can be found in Appendix A.

Original: Experimental details: We train separate hyper-representations on each of the model zoos.
Revised: Experimental details: We train separate hyper-representations on each of the model zoos.

Original: Images and labels are not used to train the hyper-representations (see § 2).
Revised: Images and labels are not used to train the hyper-representations (see § 2).

Original: Using the proposed sampling methods (§ 3.2), we generate new embeddings and decode them to weights.
Revised: Using the proposed sampling methods (§ 3.2), we generate new embeddings and decode them to weights.

Original: We evaluate sampled populations as initializations (epoch 0) and by fine-tuning for up to 25 epochs.
Revised: We evaluate sampled populations as initializations (epoch 0) and by ﬁne-tuning for up to 25 epochs.

Original: We distinguish between in-dataset and transfer-learning.
Revised: We distinguish between in-dataset and transfer-learning.

Original: For in-dataset, the same image dataset is used for training and evaluating our hyper-representations and baselines.
Revised: For in-dataset, the same image dataset is used for training and evaluating our hyper-representations and baselines.

Original: For transfer-learning, hyper-representations (and pre-trained models in baselines) are trained on a source dataset, then all populations are evaluated and fine-tuned on a different target dataset.
Revised: For transfer-learning, hyper-representations (and pre-trained models in baselines) are trained on a source dataset, then all populations are evaluated and ﬁne-tuned on a different target dataset.

Original: Full details on training, including infrastructure and compute is detailed in the Appendix B.
Revised: Full details on training, including infrastructure and compute is detailed in the Appendix B.

Original: Baselines: As the first baseline, we consider the autoencoder of [37], which is same as ours but without the proposed layer-wise loss-normalization (LWLN, § 3.1).
Revised: Baselines: As the ﬁrst baseline, we consider the autoencoder of [37], which is same as ours but without the proposed layer-wise loss-normalization (LWLN, § 3.1).

Original: We combine this autoencoder with the S KDE30 sampling method and, hence, denote it as B KDE 30 .
Revised: We combine this autoencoder with the S KDE30 sampling method and, hence, denote it as B KDE 30 .

Original: We consider two other baselines based on training models with stochastic gradient descent (SGD): training from scratch on the target classification task B T , and training on a source followed by fine-tuning on the target task B F .
Revised: We consider two other baselines based on training models with stochastic gradient descent (SGD): training from scratch on the target classiﬁcation task B T , and training on a source followed by ﬁne-tuning on the target task B F .

Original: The latter remains one of the strongest transfer learning baselines [4, 9, 22].
Revised: The latter remains one of the strongest transfer learning baselines [4, 9, 22].

Original: Reproducibility, reliability and comparability: We compare populations of at least 50 models to evaluate each method reliably.
Revised: Reproducibility, reliability and comparability: We compare populations of at least 50 models to evaluate each method reliably.

Original: We report standard deviation in Tables 1-2 and statistical significance, effect size and 95% confidence interval in Appendix F. To ensure fairness and comparability, all methods share training hyperparameters.
Revised: We report standard deviation in Tables 1-2 and statistical signiﬁcance, effect size and 95% conﬁdence interval in Appendix F. To ensure fairness and comparability, all methods share training hyperparameters.

Original: Fine-tuning uses the hyperparameters of the target domain.
Revised: Fine-tuning uses the hyperparameters of the target domain.

Original: 4.2 Results
Revised: 4.2 Results

Original: In the following, we first analyze the learned hyper-representations further justifying our sampling methods and assumptions made in § 3.2.
Revised: In the following, we ﬁrst analyze the learned hyper-representations further justifying our sampling methods and assumptions made in § 3.2.

Original: We then confirm the effectiveness of our approach for model initialization without and with fine-tuning in the in-dataset and transfer learning settings.
Revised: We then conﬁrm the effectiveness of our approach for model initialization without and with ﬁne-tuning in the in-dataset and transfer learning settings.

Original: 4.2.1 Hyper-Representations are Robust and Smooth
Revised: 4.2.1 Hyper-Representations are Robust and Smooth

Original: We evaluate the robustness and smoothness of the hyper-representation space with two experiments on the SVHN zoo.
Revised: We evaluate the robustness and smoothness of the hyper-representation space with two experiments on the SVHN zoo.

Original: First, to evaluate robustness, we add different levels of noise to the embeddings of the test set to create ˜ z , decode them to model weights ˜ w and compute models’ accuracies on the
Revised: First, to evaluate robustness, we add different levels of noise to the embeddings of the test set to create ˜ z , decode them to model weights ˜ w and compute models’ accuracies on the

Original: SVHN classification task.
Revised: SVHN classiﬁcation task.

Original: We found that both the baseline as well as our hyper-representations are robust to noise as large levels of relative noise >10% are required to affect performance (Figure 3, a,c).
Revised: We found that both the baseline as well as our hyper-representations are robust to noise as large levels of relative noise >10% are required to affect performance (Figure 3, a,c).

Original: Second, to probe for smoothness, we linearly interpolate between the test set embeddings (i) along
Revised: Second, to probe for smoothness, we linearly interpolate between the test set embeddings (i) along

Original: [Figure]
Revised: [Figure]

Original: the trajectory of the same model atdifferent epochs (z ep and z,ep 25 ) and (ii) between 250 random pairs of embeddings on the trajectories of different models ( z i and z j ).
Revised: the trajectory of the same model at epochs ( and z and (ii) between 250 random pairs of embeddings on the trajectories of different models ( z i and z j ).

Original: We decode the interpolated embeddings and compute models’ accuracies on the classification task.
Revised: We decode the interpolated embeddings and compute models’ accuracies on the classiﬁcation task.

Original: For our model, we found remarkably smooth development of accuracy along the interpolation in both schemes (Figure 3, d).
Revised: For our model, we found remarkably smooth development of accuracy along the interpolation in both schemes (Figure 3, d).

Original: The lack of fluctuations along and between trajectories support both local and global notions of smoothness in hyper-representation space.
Revised: The lack of ﬂuctuations along and between trajectories support both local and global notions of smoothness in hyper-representation space.

Original: For the baseline autoencoder (without LWLN) decoded models all perform close to 10% accuracy, so these representations do not support similar notions of smoothness (Figure 3, b), while robustness can be misleading, since the accuracy even without adding noise is already low (Figure 3, a).
Revised: For the baseline autoencoder (without LWLN) decoded models all perform close to 10% accuracy, so these representations do not support similar notions of smoothness (Figure 3, b), while robustness can be misleading, since the accuracy even without adding noise is already low (Figure 3, a).

Original: Therefore,
Revised: Therefore,

Original: LWLN together with regularizations added to the autoencoder allow for learning robust and smooth hyper-representation.
Revised: LWLN together with regularizations added to the autoencoder allow for learning robust and smooth hyper-representation.

Original: This property makes sampling from that representation more meaningful as we show next.
Revised: This property makes sampling from that representation more meaningful as we show next.

Original: 4.2.2 Sampling for In-dataset Initialization
Revised: 4.2.2 Sampling for In-dataset Initialization

Original: Comparison between sampling methods: We evaluate the performance of different sampled populations (obtained with LWLN) without fine-tuning generated weights.
Revised: Comparison between sampling methods: We evaluate the performance of different sampled populations (obtained with LWLN) without ﬁne-tuning generated weights.

Original: On MNIST, all sampled models except those obtained using S U and S C perform better than random initialization (10% accuracy), but worse than models trained from scratch B T for 25 epochs (Figure 4).
Revised: On MNIST, all sampled models except those obtained using S U and S C perform better than random initialization (10% accuracy), but worse than models trained from scratch B T for 25 epochs (Figure 4).

Original: Distribution-based samples ( S KDE and S GAN ) perform better than neighborhood based samples ( S Neigh ).
Revised: Distribution-based samples ( S KDE and S GAN ) perform better than neighborhood based samples ( S Neigh ).

Original: The populations based on the top 30% perform better than their 100% counterparts with S KDE30 as the strongest sampling method overall.
Revised: The populations based on the top 30% perform better than their 100% counterparts with S KDE30 as the strongest sampling method overall.

Original: This demonstrates that the learned hyper-representation and sampling methods are able to capture complex subtleties in weight space differentiating high and low performing models.
Revised: This demonstrates that the learned hyper-representation and sampling methods are able to capture complex subtleties in weight space differentiating high and low performing models.

Original: Comparison to the baseline hyper-representations: We also compare S KDE30 that is based on our autoencoder with layer-wise loss normalization (LWLN) to the baseline autoencoder using the same sampling method ( B KDE 30 ) without fine-tuning.
Revised: Comparison to the baseline hyper-representations: We also compare S KDE30 that is based on our autoencoder with layer-wise loss normalization (LWLN) to the baseline autoencoder using the same sampling method ( B KDE 30 ) without ﬁne-tuning.

Original: On all datasets except for MNIST, S KDE30 considerably outperform B KDE 30 with the latter performing just above 10% (random guessing), see Table(rows with epoch 0).
Revised: On all datasets except for MNIST, S KDE30 considerably outperform B KDE 30 with the latter performing just above 10% (random guessing), see Table(rows with epoch 0).

Original: We attribute the success of LWLN to two main factors.
Revised: We attribute the success of LWLN to two main factors.

Original: First, LWLN prevents the collapse of reconstruction to the mean (compare Figure 2 top to bottom).
Revised: First, LWLN prevents the collapse of reconstruction to the mean (compare Figure 2 top to bottom).

Original: Second, by fixing the weak links, the reconstructed models perform significantly better (see Appendix C for more results).
Revised: Second, by ﬁxing the weak links, the reconstructed models perform signiﬁcantly better (see Appendix C for more results).

Original: In-dataset fine-tuning: When fine-tuning, our S KDE 30 and baseline B KDE 30 appear to gradually converge to similar performance (Table 1).
Revised: In-dataset ﬁne-tuning: When ﬁne-tuning, our S KDE 30 and baseline B KDE 30 appear to gradually converge to similar performance (Table 1).

Original: While unfortunate, this result aligns well with previous findings that longer training and enough data make initialization less important [30, 17, 34].
Revised: While unfortunate, this result aligns well with previous ﬁndings that longer training and enough data make initialization less important [30, 17, 34].

Original: [Table]
Revised: [Table]

Original: We also compare S KDE 30 and B KDE 30 to training models from scratch ( B T ).
Revised: We also compare S KDE 30 and B KDE 30 to training models from scratch ( B T ).

Original: On all four datasets, both ours and the baseline hyper-representations outperform B T when generated weights are fine-tuned for the same number of epochs as B T .
Revised: On all four datasets, both ours and the baseline hyper-representations outperform B T when generated weights are ﬁne-tuned for the same number of epochs as B T .

Original: Notably, on MNIST and SVHN generated weights fine-tuned for 25 epochs are even better than B T run for 50 epochs.
Revised: Notably, on MNIST and SVHN generated weights ﬁne-tuned for 25 epochs are even better than B T run for 50 epochs.

Original: Comparison to 50 epochs is more fair though, since the hyper-representations were trained on model weights trained for up to 25 epochs.
Revised: Comparison to 50 epochs is more fair though, since the hyper-representations were trained on model weights trained for up to 25 epochs.

Original: These findings show that the models initialized with generated weights learn faster achieving better results in 25 epochs than
Revised: These ﬁndings show that the models initialized with generated weights learn faster achieving better results in 25 epochs than

Original: B T in 50 epochs.
Revised: B T in 50 epochs.

Original: Sampling ensembles: We found that a potentially useful byproduct of learning hyper-representations is the ability to generate high-performant ensembles at almost no extra computational cost, since both sampling and generation are computationally cheap.
Revised: Sampling ensembles: We found that a potentially useful byproduct of learning hyper-representations is the ability to generate high-performant ensembles at almost no extra computational cost, since both sampling and generation are computationally cheap.

Original: To demonstrate this effect, we compare ensembles formed using the baseline autoencoder ( B KDE 30 ) and ours ( S KDE 30 ) to the ensembles composed of models trained from scratch for 25 epochs ( B T ) on
Revised: To demonstrate this effect, we compare ensembles formed using the baseline autoencoder ( B KDE 30 ) and ours ( S KDE 30 ) to the ensembles composed of models trained from scratch for 25 epochs ( B T ) on

Original: SVHN.
Revised: SVHN.

Original: Ensembles generated using the baseline B KDE 30 stagnate below 20% (Figure 5).
Revised: Ensembles generated using the baseline B KDE 30 stagnate below 20% (Figure 5).

Original: In contrast, ensembles generated using our
Revised: In contrast, ensembles generated using our

Original: S KDE 30 gracefully improve with the ensemble size outperforming single B T models and almost matching B T ensembles with enough models in the ensembles.
Revised: S KDE 30 gracefully improve with the ensemble size outperforming single B T models and almost matching B T ensembles with enough models in the ensembles.

Original: Remarkably, the average test accuracy of generated ensembles of 15 models is 77.6%, which is considerably higher than 70.7% of models trained on SVHN for 50 epochs.
Revised: Remarkably, the average test accuracy of generated ensembles of 15 models is 77.6%, which is considerably higher than 70.7% of models trained on SVHN for 50 epochs.

Original: We conclude that hyper-representations learned with LWLN generate models that are not only performant, but also diverse.
Revised: We conclude that hyper-representations learned with LWLN generate models that are not only performant, but also diverse.

Original: Although generating ensembles requires learning hyper-representation and model zoo first, we assume that in future such a hyper-representation can be trained once and reused in unseen scenarios as we tentatively explore below (see results in Table 3 and the discussion therein).
Revised: Although generating ensembles requires learning hyper-representation and model zoo ﬁrst, we assume that in future such a hyper-representation can be trained once and reused in unseen scenarios as we tentatively explore below (see results in Table 3 and the discussion therein).

Original: Do reconstructed models become similar to the original during fine-tuning?
Revised: Do reconstructed models become similar to the original during ﬁne-tuning?

Original: Sampled hyperrepresentations often learn faster and to a higher performance than the population of models they were trained on (Table 1).
Revised: Sampled hyperrepresentations often learn faster and to a higher performance than the population of models they were trained on (Table 1).

Original: We therefore explore the question, if reconstructed models develop in weight space in the same direction as their original, or find a different solution.
Revised: We therefore explore the question, if reconstructed models develop in weight space in the same direction as their original, or ﬁnd a different solution.

Original: On SVHN, we found that the reconstructed models ( ˆ w ) after one epoch of fine-tuning perform similar to their originals ( w ) and slightly outperform from there on (Figure 6, left).
Revised: On SVHN, we found that the reconstructed models ( ˆ w ) after one epoch of ﬁne-tuning perform similar to their originals ( w ) and slightly outperform from there on (Figure 6, left).

Original: At the same time, pairs of original and reconstructed models move further apart and become less aligned in weight space (Figure 6, right).
Revised: At the same time, pairs of original and reconstructed models move further apart and become less aligned in weight space (Figure 6, right).

Original: It appears that reconstructed models perform better and explore different solutions in weight space to do so.
Revised: It appears that reconstructed models perform better and explore different solutions in weight space to do so.

Original: This confirms the intuition that hyper-representations impress useful structure on decoded weights.
Revised: This conﬁrms the intuition that hyper-representations impress useful structure on decoded weights.

Original: A pass through encoder and decoder thus results not just in a noisy reconstruction of the original sample.
Revised: A pass through encoder and decoder thus results not just in a noisy reconstruction of the original sample.

Original: Instead, it maps to a different region on the loss surface, which leads to faster learning and better solutions.
Revised: Instead, it maps to a different region on the loss surface, which leads to faster learning and better solutions.

Original: Combining this with the ensembling results in Figure 5, hyper-representations do not collapse to a single solution, but decode to diverse and useful weights.
Revised: Combining this with the ensembling results in Figure 5, hyper-representations do not collapse to a single solution, but decode to diverse and useful weights.

Original: 4.2.3 Sampling Initializations for Transfer Learning
Revised: 4.2.3 Sampling Initializations for Transfer Learning

Original: Setup: We investigate the effectiveness of our method in a transfer-learning setup across image datasets.
Revised: Setup: We investigate the effectiveness of our method in a transfer-learning setup across image datasets.

Original: In particular, we report transfer learning results from SVHN to MNIST and from STLto CIFAR-10 as two representative scenarios.
Revised: In particular, we report transfer learning results from SVHN to MNIST and from STLto CIFAR-10 as two representative scenarios.

Original: Results on the other pairs of datasets can be found in Appendix F.
Revised: Results on the other pairs of datasets can be found in Appendix F.

Original: In these experiments, pre-trained models B F and the hyper-representation model are trained on a source domain.
Revised: In these experiments, pre-trained models B F and the hyper-representation model are trained on a source domain.

Original: Subsequently, the pre-trained models B F and the samples S KDE ,
Revised: Subsequently, the pre-trained models B F and the samples S KDE ,

Original: S Neigh and S GAN are fine-tuned on the target domain.
Revised: S Neigh and S GAN are ﬁne-tuned on the target domain.

Original: The baseline approach ( B T ) is based on training models from scratch on the target domain.
Revised: The baseline approach ( B T ) is based on training models from scratch on the target domain.

Original: [Table]
Revised: [Table]

Original: Results: When transfer learning is performed from SVHN to MNIST, the sampled populations on average learn faster and achieve significantly higher performance than the B T baseline and generally compares favorably to B F (Figure 7, Table 2).
Revised: Results: When transfer learning is performed from SVHN to MNIST, the sampled populations on average learn faster and achieve signiﬁcantly higher performance than the B T baseline and generally compares favorably to B F (Figure 7, Table 2).

Original: In the STL-10 to CIFAR-10 experiment, all populations appear to saturate with only small differences in their performances (Table 2).
Revised: In the STL-10 to CIFAR-10 experiment, all populations appear to saturate with only small differences in their performances (Table 2).

Original: Different sampling methods perform differently at the beginning versus the end of transfer learning.
Revised: Different sampling methods perform differently at the beginning versus the end of transfer learning.

Original: Generally,
Revised: Generally,

Original: S KDE30 performs better in the first epochs, while all methods perform comparably at the end of transfer-learning.
Revised: S KDE30 performs better in the ﬁrst epochs, while all methods perform comparably at the end of transfer-learning.

Original: These discrepancies underline the difficulty of developing a single strong sampling method, which is an interesting area of future research.
Revised: These discrepancies underline the difﬁculty of developing a single strong sampling method, which is an interesting area of future research.

Original: We further found that all datasets are useful sources for all targets (see Appendix F).
Revised: We further found that all datasets are useful sources for all targets (see Appendix F).

Original: Interestingly and other than in related work [29], even transfer from the simpler to harder datasets (e.g., MNIST to SVHN) improves performance.
Revised: Interestingly and other than in related work [29], even transfer from the simpler to harder datasets (e.g., MNIST to SVHN) improves performance.

Original: This might be explained by the ability of hyper-representations to capture a generic inductive prior useful across different domains, which we further investigate next.
Revised: This might be explained by the ability of hyper-representations to capture a generic inductive prior useful across different domains, which we further investigate next.

Original: [Table]
Revised: [Table]

Original: Conditioning on unseen zoos: We explore if the hyper-representation trained on the models of one zoo (e.g. MNIST) can reconstruct the weights of another unseen zoo (e.g. SVHN).
Revised: Conditioning on unseen zoos: We explore if the hyper-representation trained on the models of one zoo (e.g. MNIST) can reconstruct the weights of another unseen zoo (e.g. SVHN).

Original: This can be useful to enable generation of weights for novel tasks without the need to retrain a hyper-representation.
Revised: This can be useful to enable generation of weights for novel tasks without the need to retrain a hyper-representation.

Original: This is analogous to instance-conditioned GANs that recently were able to generate images from unseen domains without retraining GANs [2].
Revised: This is analogous to instance-conditioned GANs that recently were able to generate images from unseen domains without retraining GANs [2].

Original: Our results in Table 3 show that while the performance on the unseen zoos is reduced, it is still well above random guessing (10%), especially when multiple model weights are sampled and ensembled.
Revised: Our results in Table 3 show that while the performance on the unseen zoos is reduced, it is still well above random guessing (10%), especially when multiple model weights are sampled and ensembled.

Original: This is promising, as the hyper-representations were trained on single-dataset zoos.
Revised: This is promising, as the hyper-representations were trained on single-dataset zoos.

Original: 4.2.4 Sampling Initializations for Unseen Architectures
Revised: 4.2.4 Sampling Initializations for Unseen Architectures

Original: Generalization to unseen large architectures with complex connectivity (ResNet, MobileNet, and
Revised: Generalization to unseen large architectures with complex connectivity (ResNet, MobileNet, and

Original: EfficientNet) is a very interesting and ambitious research problem.
Revised: EfﬁcientNet) is a very interesting and ambitious research problem.

Original: As a step towards that goal, we perform experiments in which we attempted to use our hyper-representation beyond the same simple architecture.
Revised: As a step towards that goal, we perform experiments in which we attempted to use our hyper-representation beyond the same simple architecture.

Original: Surprisingly, our results indicate the promise of leveraging the hyper-representation for more diverse architectures and settings.
Revised: Surprisingly, our results indicate the promise of leveraging the hyper-representation for more diverse architectures and settings.

Original: Further experiments investigating the cross-architecture generalization capabilities of hyper-representations can be found in Appendix D.
Revised: Further experiments investigating the cross-architecture generalization capabilities of hyper-representations can be found in Appendix D.

Original: Setup: With this experiment, we aim to verify if it is possible to adapt our approach to architectures not seen during training, e.g., with skip connections and/or with more layers.
Revised: Setup: With this experiment, we aim to verify if it is possible to adapt our approach to architectures not seen during training, e.g., with skip connections and/or with more layers.

Original: We follow the transferlearning setup of § 4.2.3 and use an existing MNIST hyper-representation to sample weights as initializiation for training on SVHN.
Revised: We follow the transferlearning setup of § 4.2.3 and use an existing MNIST hyper-representation to sample weights as initializiation for training on SVHN.

Original: However, we now also vary the architecture.
Revised: However, we now also vary the architecture.

Original: While the decoder outputs a fixed-sized vector of weights, we can assign these weights to new architectures by either making sure that the new architecture still has the same number of parameters or by initializing randomly the extra parameters introduced.
Revised: While the decoder outputs a ﬁxed-sized vector of weights, we can assign these weights to new architectures by either making sure that the new architecture still has the same number of parameters or by initializing randomly the extra parameters introduced.

Original: Specifically, we create three cases: (1) we add ResNet-style skip connections [16] (1x1 conv) to the convolutional layers (3-conv + res-skip), (2) re-distribute the weights to smaller four convolutional layers (4-conv), (3) re-distribute to smaller four convolutional layers and add identity skip connections (4-conv + id skip
Revised: Speciﬁcally, we create three cases: (1) we add ResNet-style skip connections [16] (1x1 conv) to the convolutional layers (3-conv + res-skip), (2) re-distribute the weights to smaller four convolutional layers (4-conv), (3) re-distribute to smaller four convolutional layers and add identity skip connections (4-conv + id.-skip).

Original: [Table]
Revised: [Table]

Original: Results: Surprisingly, despite training our hyper-representation on the models of the same architecture, generated weights for all three cases outperform random initialization and converge significantly faster across all the variations (Table 4).
Revised: Results: Surprisingly, despite training our hyper-representation on the models of the same architecture, generated weights for all three cases outperform random initialization and converge signiﬁcantly faster across all the variations (Table 4).

Original: In all the variations even just after 5 epochs the models with generated weights are better than training the baseline for 50 epochs.
Revised: In all the variations even just after 5 epochs the models with generated weights are better than training the baseline for 50 epochs.

Original: In the 3-conv + res-skip experiments, some models in both populations did not learn, which leads to high standard deviation.
Revised: In the 3-conv + res-skip experiments, some models in both populations did not learn, which leads to high standard deviation.

Original: Further analysis is required to explain the gains of our approach in this challenging setup.
Revised: Further analysis is required to explain the gains of our approach in this challenging setup.

Original: To extend and scale up our method further, future work could combined it with the methods of growing networks [3, 42], so that some layers are generated while some are initialized in a sophisticated way to preserve the functional form of the network.
Revised: To extend and scale up our method further, future work could combined it with the methods of growing networks [3, 42], so that some layers are generated while some are initialized in a sophisticated way to preserve the functional form of the network.

Original: 4.3 Limitations of Zoos with Small Models
Revised: 4.3 Limitations of Zoos with Small Models

Original: To thoroughly investigate different methods and make experiments feasible, we chose to use the model zoos of the same small scale as in [37].
Revised: To thoroughly investigate different methods and make experiments feasible, we chose to use the model zoos of the same small scale as in [37].

Original: While on MNIST and SVHN, the architectures of such model zoos allowed us to achieve high performance, on CIFAR-10 and STL-10, the performance of all populations is limited by the low capacity of the models zoo’s architecture.
Revised: While on MNIST and SVHN, the architectures of such model zoos allowed us to achieve high performance, on CIFAR-10 and STL-10, the performance of all populations is limited by the low capacity of the models zoo’s architecture.

Original: The models saturate at around 50% and 40% accuracy, respectively.
Revised: The models saturate at around 50% and 40% accuracy, respectively.

Original: The sampled populations reach the saturation point and fluctuate, but cannot outperform the baselines, see Appendix F for details.
Revised: The sampled populations reach the saturation point and ﬂuctuate, but cannot outperform the baselines, see Appendix F for details.

Original: We hypothesize that due to the high remaining loss, the weight updates are correspondingly large without converging or improving performance.
Revised: We hypothesize that due to the high remaining loss, the weight updates are correspondingly large without converging or improving performance.

Original: This may cause the weights to contain relatively little signal and high noise.
Revised: This may cause the weights to contain relatively little signal and high noise.

Original: Larger model architectures might mitigate this behaviour.
Revised: Larger model architectures might mitigate this behaviour.

Original: Corresponding model zoos have recently been made available in [38] to tackle this issue 1 .
Revised: Corresponding model zoos have recently been made available in [38] to tackle this issue 1 .

Original:  Related Work
Revised:  Related Work

Original: HyperNetworks:
Revised: HyperNetworks:

Original: Recently, representation learning on neural networks is typically based on
Revised: Recently, representation learning on neural networks is typically based on

Original: HyperNetworks that learn low-dimensional structure of model weights to generate weights in a deterministic fashion [14, 1, 21, 45].
Revised: HyperNetworks that learn low-dimensional structure of model weights to generate weights in a deterministic fashion [14, 1, 21, 45].

Original: HyperNetworks have also been extended to meta-learning by conditioning weight generation on data [46, 36].
Revised: HyperNetworks have also been extended to meta-learning by conditioning weight generation on data [46, 36].

Original: Closely related to our work, HyperGANs [35] can sample model weights by combining the hypernetworks and the GAN framework.
Revised: Closely related to our work, HyperGANs [35] can sample model weights by combining the hypernetworks and the GAN framework.

Original: Similarly, [8] allow for sampling model weights by conditioning the hypernetwork on a noise vector.
Revised: Similarly, [8] allow for sampling model weights by conditioning the hypernetwork on a noise vector.

Original: However, training hypernetwork-based methods require input data (e.g. images) to feed to the neural networks.
Revised: However, training hypernetwork-based methods require input data (e.g. images) to feed to the neural networks.

Original: In practice, there may already be large collections of trained models, while their training data may not always be accessible.
Revised: In practice, there may already be large collections of trained models, while their training data may not always be accessible.

Original: Learning representations of model weights without data, called hyper-representations, has been recently introduced in [37].
Revised: Learning representations of model weights without data, called hyper-representations, has been recently introduced in [37].

Original: Our methods build on that work to allow for better reconstruction and sampling.
Revised: Our methods build on that work to allow for better reconstruction and sampling.

Original: [7] showed that given a few parameters of a network, the remaining values of a single model can be accurately reconstructed.
Revised: [7] showed that given a few parameters of a network, the remaining values of a single model can be accurately reconstructed.

Original: However, in our work we leverage the autoencoder to train a representation of the entire model zoo.
Revised: However, in our work we leverage the autoencoder to train a representation of the entire model zoo.

Original: Very recently, [33] use diffusion on a population of models to generate model weights for the original task via prompting.
Revised: Very recently, [33] use diffusion on a population of models to generate model weights for the original task via prompting.

Original: Transfer Learning: Transfer learning via fine-tuning aims at re-using models and their learned knowledge from a source to a target task [44, 4, 9, 29, 22].
Revised: Transfer Learning: Transfer learning via ﬁne-tuning aims at re-using models and their learned knowledge from a source to a target task [44, 4, 9, 29, 22].

Original: Transfer learning models makes training less expensive, boosts performance, or allows to train on datasets with very few samples and has been applied on a wide range of domains [48].
Revised: Transfer learning models makes training less expensive, boosts performance, or allows to train on datasets with very few samples and has been applied on a wide range of domains [48].

Original: The common transfer learning methods however only consider transferring from a single model, and so disregard the large variety of pre-trained models and potential benefit of combining them.
Revised: The common transfer learning methods however only consider transferring from a single model, and so disregard the large variety of pre-trained models and potential beneﬁt of combining them.

Original: Knowledge distillation: Our work is related to [43, 25, 39] that allow to distill knowledge from a model zoo into a single network.
Revised: Knowledge distillation: Our work is related to [43, 25, 39] that allow to distill knowledge from a model zoo into a single network.

Original: Knowledge distillation overcomes the inherent limitation of transfer learning by transferring the knowledge from many large teacher models to a relatively small student model [25, 39].
Revised: Knowledge distillation overcomes the inherent limitation of transfer learning by transferring the knowledge from many large teacher models to a relatively small student model [25, 39].

Original: Knowledge distillation however requires the source models at training as in [25] and at inference as in[39] thus increasing memory cost.
Revised: Knowledge distillation however requires the source models at training as in [25] and at inference as in[39] thus increasing memory cost.

Original: Further, the learned knowledge cannot be shared between different target models.
Revised: Further, the learned knowledge cannot be shared between different target models.

Original: Learnable initialization [6, 47] provide methods to improve initialization by leveraging the meta-learning and gradient-flow ideas.
Revised: Learnable initialization [6, 47] provide methods to improve initialization by leveraging the meta-learning and gradient-ﬂow ideas.

Original: In contrast to knowledge distillation and learnable initialization, we train a hyper-representation of a model zoo in a latent space, which is a more general and powerful approach that can enable sampling an ensemble, property estimation, improved initialization and implicit knowledge distillation across datasets.
Revised: In contrast to knowledge distillation and learnable initialization, we train a hyper-representation of a model zoo in a latent space, which is a more general and powerful approach that can enable sampling an ensemble, property estimation, improved initialization and implicit knowledge distillation across datasets.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose a new method to sample from hyper-representations to generate neural network weights in one forward pass.
Revised: In this paper, we propose a new method to sample from hyper-representations to generate neural network weights in one forward pass.

Original: We extend the training objective of hyper-representations by a novel layer-wise loss normalization which is key to the capability of generating functional models.
Revised: We extend the training objective of hyper-representations by a novel layer-wise loss normalization which is key to the capability of generating functional models.

Original: Our method allows us to generate diverse populations of model weights, which show high performance as ensembles.
Revised: Our method allows us to generate diverse populations of model weights, which show high performance as ensembles.

Original: We evaluate sampled models both in-dataset as well as in transfer learning and find them capable to outperform both models trained from scratch, as well as pre-trained and finetuned models.
Revised: We evaluate sampled models both in-dataset as well as in transfer learning and ﬁnd them capable to outperform both models trained from scratch, as well as pre-trained and ﬁnetuned models.

Original: Populations of sampled models, even for some unseen architectures, generally learn faster and achieve statistically significantly higher performance.
Revised: Populations of sampled models, even for some unseen architectures, generally learn faster and achieve statistically signiﬁcantly higher performance.

Original: This demonstrates that such hyperrepresentation can be used as a generative model for neural network weights and therefore might serve as a building block for transfer learning from different domains, meta learning or continual learning.
Revised: This demonstrates that such hyperrepresentation can be used as a generative model for neural network weights and therefore might serve as a building block for transfer learning from different domains, meta learning or continual learning.

Original: Acknowledgments
Revised: Acknowledgments

Original: This work was partially funded by Google Research Scholar Award, the University of
Revised: This work was partially funded by Google Research Scholar Award, the University of

Original: St.Gallen Basic Research Fund, and project PID2020-117142GB-I00 funded by MCIN/ AEI /10.13039/501100011033.
Revised: St.Gallen Basic Research Fund, and project PID2020-117142GB-I00 funded by MCIN/ AEI /10.13039/501100011033.

Original: We are thankful to Michael Mommert for editorial support.
Revised: We are thankful to Michael Mommert for editorial support.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section.
Revised: [Yes] See Section.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.3 and(c)
Revised: [Yes] See Section 4.3 and(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [No] The method we propose is a general machine learning method.
Revised: [No] The method we propose is a general machine learning method.

Original: It comes with all the risks involved, but in our opinion does not pose any particular risk.
Revised: It comes with all the risks involved, but in our opinion does not pose any particular risk.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Code, hyper-representations of selected model zoos are included in the supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] Code, hyper-representations of selected model zoos are included in the supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] All the details are specified in the Appendix A and B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] All the details are speciﬁed in the Appendix A and B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We report means and std for all experiments in the main paper and provide additional details about median, CI95 on the median, p-value and effect size in the Appendix F (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] We report means and std for all experiments in the main paper and provide additional details about median, CI95 on the median, p-value and effect size in the Appendix F (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix B 4.
Revised: [Yes] See Appendix B 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] See Section A (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] See Section A (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] We have trained new model zoos and will make them publicly available via an URL (d) Did you discuss whether
Revised: [Yes] We have trained new model zoos and will make them publicly available via an URL (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] Our experiments use publicly available image datasets, which we cite.
Revised: [N/A] Our experiments use publicly available image datasets, which we cite.

Original: The model zoos and code for hyper-representation learning was published on github and zenodo by the authors of the original paper under Creative Commons Attribution 4.0.
Revised: The model zoos and code for hyper-representation learning was published on github and zenodo by the authors of the original paper under Creative Commons Attribution 4.0.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] Our experiments rely on common computer vision datasets and do not contain personally identifiable information nor offensive content.
Revised: [N/A] Our experiments rely on common computer vision datasets and do not contain personally identiﬁable information nor offensive content.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] A Model Zoo Details [Table]
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: The model zoos are generated following the method of [37, 38]

Original: An overview of the model zoos is given in in Table 5.

Original: All model zoos share one general CNN architecture, outlined in Table 6.

Original: The hyperparameter choices for each of the population are listed in Table 7.

Original: The hyperparameters are chosen to generate zoos with smooth, continuous development and spread in performance.

Original: [Table]

Original: [Table]

Original: Hyper-representations are learned with an autoencoder based on multi-head self-attention.

Original: The architecture is outlined in Figure 8.

Original: Convolutional and fully connected neurons are embedded to token embeddings of dimension d token .

Original: Learned position encodings are added to provide relational information.

Original: A learned compression token (CLS) is appended to the sequence of token embeddings.

Original: The sequence of token embeddings is passed to N layers layers of multi-head self-attention with

Original: N heads heads with hidden embedding dimension d hidden .

Original: The CLS token is compressed to the bottleneck of dimension d z with an MLP or a linear layer.

Original: For the decoder, an MLP or a linear layer maps the bottleneck to a sequence of token embeddings.

Original: The sequence is passed through another stack of multi-head self-attention, which is symmetric to the encoder.

Original: Debedders map the token embeddings back to convolutional and fully connected neurons.

Original: The reconstruction and contrastive loss are balanced with a parameter β .

Original: The contrastive loss is computed on the embeddings z mapped through a projection head ¯z =

Original: p ( z , where p is a learned MLP with four layers with 400 neurons each and ¯z has 50 dimensions.

Original: In Table 8, the exact hyper-parameters for each of the hyper-representation are listed to reproduce our results.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: C Evaluation of Layer-Wise Loss Normalization

Original: To evaluate layer-wise loss normalization, we compare two hyper-representations with comparable reconstruction.

Original: Both have a R 2 = 1 − mse (ˆ w , w ) mse ( w mean , w as a measure of the explained variance of around 70%.

Original: One is trained trained with the baseline hyper-representation MSE, the other with layer-wise-normalization.

Original: Figures 9 and 10 show the distribution of weights per layer before and after reconstruction, as well as the accuracy distribution of populations on the SVHN image test set.

Original: With the basline learning scheme in Figure 9, the distributions in layers 3 and 4 do not match.

Original: In these layers, the original weight distribution is smaller, and so there is only a small error even if the reconstructions predicts the mean.

Original: These layers become a weak link of the reconstructed models, and cause performance around random guessing.

Original: With layer-wise loss normalization in Figure 10, the weight distribution between the layers becomes more similar.

Original: As a consequence, the reconstruction error is more evenly distributed across the layers, there are no single layers that aren’t reconstructed at all.

Original: This appears to allow information to flow forward through the model, and significantly improves the performance of reconstructed models.

Original: We find layer-wise-normalization necessary to reconstruct or sample functional models across all populations, where the weights are unevenly distributed.

Original: D Hyper-Representation Analysis

Original: In this section, we detail the analysis of hyper-representations.

Original: We begin with their geometry, followed by the distributions of individual dimensions of hyper-representations, and finally investigate robustness and smoothness.

Original: Embeddings in Hyper-Representation Space Populate a Hyper-Sphere

Original: We analyse the geometry of hyper-representations z .

Original: The space of hyper-representations is bounded to a high dimensional box by a tanh activation.

Original: Surprisingly, hyper-representations do not populate the entire space, but sections on a shell of a high-dimensional sphere.

Original: Figure 11 shows the distribution of the norm of the embeddings of the MNIST zoo.

Original: All embeddings are distributed on a small band between lengthand 12, therefore they must populate the shell of a hyper-sphere.

Original: In Figure 12 we investigate pairwise cosine distances between the embeddings of the MNIST zoo.

Original: The majority of the embeddings populate the region between 0.6 and 0.8.

Original: The outliers around 1.0 are embeddings of the same model at different epochs.

Original: This indicates that models are not entirely orthogonal, but mutually equally far apart, populating a section of the shell of the hyper-sphere.

Original: While hyper-spheres are commonly found in embeddings of contrastive learning [19], in our experiments hyper-spheres form even without a contrastive loss.

Original: Properties of the models embedded on that hyper-sphere can be predicted from hyper-representations, therefore the topology on the sphere appears to encode model properties.

Original: Distributions of Dimensions of Embeddings in Hyper-Representation Encode Properties Previous work showed that linear probing from hyper-representations accurately predicts i.e. model accuracy.

Original: In these linear probes, the individual z dimensions each linearly contribute to accuracy predictions.

Original: This allows us investigate z dimensions independently.

Original: Figure 13 shows examples for the distribution of selected individual dimensions of hyper-representations z .

Original: On the left are the distribution of the entire population, on the right of the top 30 % performing models.

Original: The individual dimensions show different types of distributions, with different modes.

Original: Most have a zero mean and span 3/4 of the available range, but some collapse to either − 1 or 1 .

Original: Further, the distributions also differ in at least some dimension between the entire population, and the better performing split of the population.

Original: Generalization Capabilities of Hyper-Representations to Diverse Model Zoos There are certain architectural changes such as adding/removing/changing pooling layers and nonlinearity that do not change the number of parameters (the dimensionality of the input/output required by our approach).

Original: These changes as well as changes of hyperparameters used to train models in a zoo may drastically alter the distribution of weights and pose a challenge to the proposed approach.

Original: Modern neural networks (ResNet, MobileNet, EfficientNet, etc.) are often trained with very different hyperparameters.

Original: With the experiment below, we investigate the generalization capabilities of hyper-representations to suchchanges, which might be important for modern large-scale settings as well.

Original: Setup: We experimentally evaluate generalizability of the proposed approach on models trained with a different choice of nonlinearity or other hyperparameters with two experiments (a and b).

Original: To that end, in addition to the original SVHN test zoo (zoo 1), we use two more diverse SVHN zoos (zoo 2 and zoo 3).

Original: In zoo 2, in addition to random seed, models differ in the activation (tanh, relu, gelu, sigmoid), l2-regularization (0, 0.001, 0.1) and dropout (0,0.3,0.5).

Original: In zoo 3 (extending zoo 2), we increase the diversity further by additionally varying the initialization method (uniform, normal, kaiming-uniform, kaiming-normal) and the learning rate (0.0001, 0.001, 0.01).

Original: Experiment (a): We first evaluate our original encoder-decoder trained on a model zoo varying in random seed only.

Original: For evaluation, we pass the test splits of zoo 2 and zoo 3 through the encoderdecoder.

Original: We measure the reconstruction R 2 score of the original encoder-decoder on the diverse test zoos.

Original: Results: Our results (Table D) indicate that our original encoder-decoder can still encode and decode weights even in such a challenging setting, although there is an expected drop of performance.

Original: Experiment (a): We next evaluate if hyper-representations can be trained on diverse zoos.

Original: For this experiment, we train a hyper-representation on the train split of zoo 3.

Original: With this, we aim to show that training hyper-representations on diverse zoos improves generalization capabilities further.

Original: Results: Our results show that training on diverse zoos is a much more difficult task to optimize, hence the reconstruction on the original zoo degrades.

Original: It nonetheless improves the reconstruction results on the test split of the diverse zoos 2 and 3.

Original: This indicates that varying seeds and hyperparameters may be different aspects of complexity that need to be considered.

Original: [Table]
Revised: [N/A]

Original: E Sampling Methods E.1 VAE

Original: A common extension of the autoencoder of [37] to enable sampling from its latent representation is to make the autoencoder variational [20].

Original: In our experiments, VAEs could not be trained to satisfactory reconstruct model weights without unweighting the KL-divergence to insignificance essentially making it deterministic as in [37].

Original: Empirically, embeddings in hyper-representations are mapped on the shell of a sphere (see Section D) and leave the inside of the sphere entirely empty.

Original: On the other hand, a gaussian prior allocates most of the probability mass near the center of the sphere.

Original: It therefore appears plausible that the two may be incompatible.

Original: That issue of non-compatible priors is well known.

Original: [10] find that regularizing embeddings and decoder yields equally smooth representation spaces as VAEs without restrictions to specific priors.

Original: During training of hyper-representations, both encoder and decoder are regularized with a small ℓ 2 penalty.

Original: Further, dropout is applied throughout the autoencoder, which servers as another regularizer and adds blurryness to the embeddings.

Original: The combination of dropout, the erasing augmentation and the contrastive loss further regularizes the hyper-representation space.

Original: In all our sampling methods, we draw samples from probability distributions, which effectively disconnects the drawn samples from training embeddings.

Original: E.2 Latent Space GAN Details

Original: The generator and discriminator of our GAN consist of four fully-connected layers interleaved with

Original: ReLU nonlinearities.

Original: The same architecture and training hyperparameters are used for all experiments.

Original: The generator’s input is a Gaussian noise n ∗ of dimensionality d = 16 , the hidden dimensionalities are 128, 256 and 512, and the output dimensionality is equal to the hyper-representation length D .

Original: The discriminator’s input is D -dimensional, the hidden dimensionalities are 1024, 512 and 256, and the output dimensionality is a scalar denoting either a real or fake sample.

Original: The discriminator is regularized with Spectral Norm [31].

Original: The discriminator and generator are trained for 1000 epochs and batch size 32 using Adam with a two time-scale update rule [

Original: F Full Experiment Results F.1 Digit Domain

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]