Original: A NALYZING P OPULATIONS OF N EURAL N ETWORKS VIA D YNAMICAL M ODEL E MBEDDING
Revised: A NALYZING P OPULATIONS OF N EURAL N ETWORKS VIA D YNAMICAL M ODEL E MBEDDING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A core challenge in the interpretation of deep neural networks is identifying commonalities between the underlying algorithms implemented by distinct networks trained for the same task.
Revised: A core challenge in the interpretation of deep neural networks is identifying commonalities between the underlying algorithms implemented by distinct networks trained for the same task.

Original: Motivated by this problem, we introduce D YNAMO , an algorithm that constructs low-dimensional manifolds where each point corresponds to a neural network model, and two points are nearby if the corresponding neural networks enact similar high-level computational processes.
Revised: Motivated by this problem, we introduce D YNAMO , an algorithm that constructs low-dimensional manifolds where each point corresponds to a neural network model, and two points are nearby if the corresponding neural networks enact similar high-level computational processes.

Original: D YNAMO takes as input a collection of pre-trained neural networks and outputs a meta-model that emulates the dynamics of the hidden states as well as the outputs of any model in the collection.
Revised: D YNAMO takes as input a collection of pre-trained neural networks and outputs a meta-model that emulates the dynamics of the hidden states as well as the outputs of any model in the collection.

Original: The speciﬁc model to be emulated is determined by a model embedding vector that the meta-model takes as input; these model embedding vectors constitute a manifold corresponding to the given population of models.
Revised: The speciﬁc model to be emulated is determined by a model embedding vector that the meta-model takes as input; these model embedding vectors constitute a manifold corresponding to the given population of models.

Original: We apply
Revised: We apply

Original: D YNAMO to both RNNs and CNNs, and ﬁnd that the resulting model embedding spaces enable novel applications: clustering of neural networks on the basis of their high-level computational processes in a manner that is less sensitive to reparameterization; model averaging of several neural networks trained on the same task to arrive at a new, operable neural network with similar task performance; and semi-supervised learning via optimization on the model embedding space.
Revised: D YNAMO to both RNNs and CNNs, and ﬁnd that the resulting model embedding spaces enable novel applications: clustering of neural networks on the basis of their high-level computational processes in a manner that is less sensitive to reparameterization; model averaging of several neural networks trained on the same task to arrive at a new, operable neural network with similar task performance; and semi-supervised learning via optimization on the model embedding space.

Original: Using a ﬁxed-point analysis of meta-models trained on populations of RNNs, we gain new insights into how similarities of the topology of RNN dynamics correspond to similarities of their high-level computational processes.
Revised: Using a ﬁxed-point analysis of meta-models trained on populations of RNNs, we gain new insights into how similarities of the topology of RNN dynamics correspond to similarities of their high-level computational processes.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A crucial feature of neural networks with ﬁxed network architecture is that they form a manifold by virtue of their continuously tunable weights, which underlies their capacity to be trained by gradient descent.
Revised: A crucial feature of neural networks with ﬁxed network architecture is that they form a manifold by virtue of their continuously tunable weights, which underlies their capability to be trained by gradient descent.

Original: However, this conception of the space of neural networks is inadequate for understanding the computational processes they perform.
Revised: However, this conception of the space of neural networks is inadequate for understanding the computational processes they perform.

Original: For example, two neural networks trained to perform the same task may have vastly different weights, and yet implement the same high-level algorithms and computational processes (Maheswaranathan et al., 2019b).
Revised: For example, two neural networks trained to perform the same task may have vastly different weights, and yet implement the same high-level algorithms and computational processes (Maheswaranathan et al., 2019b).

Original: In this paper, we construct an algorithm which provides alternative parametrizations of the space of RNNs and CNNs with the goal of endowing a geometric structure that is more compatible with the high-level computational processes performed by neural networks.
Revised: In this paper, we construct an algorithm which provides alternative parametrizations of the space of RNNs and CNNs with the goal of endowing a geometric structure that is more compatible with the high-level computational processes performed by neural networks.

Original: In particular, given a set of neural networks with possibly different architectures (and possibly trained on different tasks), we ﬁnd a parametrization of a low-dimensional submanifold of neural networks which approximately interpolates between these chosen “base models”, as well as extrapolates beyond them.
Revised: In particular, given a set of neural networks with possibly different architectures (and possibly trained on different tasks), we ﬁnd a parametrization of a low-dimensional submanifold of neural networks which approximately interpolates between these chosen “base models”, as well as extrapolates beyond them.

Original: We can use such model embedding spaces to cluster neural networks and even compute model averages of neural networks.
Revised: We can use such model embedding spaces to cluster neural networks and even compute model averages of neural networks.

Original: A key feature is that two points in model embedding space are nearby if they correspond to neural networks which implement similar high-level computational processes, in a manner to be described later.
Revised: A key feature is that two points in model embedding space are nearby if they correspond to neural networks which implement similar high-level computational processes, in a manner to be described later.

Original: In this way, two neural networks may correspond to nearby points in model embedding space even if those neural networks have distinct weights or even architectures.
Revised: In this way, two neural networks may correspond to nearby points in model embedding space even if those neural networks have distinct weights or even architectures.

Original: The model embedding space is parametrized by a low-dimensional parameter θ ∈ R d , and each base model is assigned a value of θ in the space.
Revised: The model embedding space is parametrized by a low-dimensional parameter θ ∈ R d , and each base model is assigned a value of θ in the space.

Original: This allows us to apply traditional ideas from clustering and interpolation to the space of neural networks.
Revised: This allows us to apply traditional ideas from clustering and interpolation to the space of neural networks.

Original: Moreover, each model embedding space has an associated meta-model which, upon being given a θ , is rendered into an operable neural network.
Revised: Moreover, each model embedding space has an associated meta-model which, upon being given a θ , is rendered into an operable neural network.

Original: If a base model is mapped to some θ , then the meta-model, upon being given that θ , will emulate the corresponding base model.
Revised: If a base model is mapped to some θ , then the meta-model, upon being given that θ , will emulate the corresponding base model.

Revised: See Figure 1 for a diagrammatic depiction.

Original: An interesting application is that given two models assigned to parameters θ 

Revised: An interesting application is that given two models assigned to parameters θ

Original: and θ 2 , we can consider the averaged model corresponding to the value ( θ 1 + θ 2 ) / 2 .
Revised: and θ 2 , we can consider the averaged model corresponding to the value ( θ 1 + θ 2 ) / 2 .

Original: We ﬁnd that this averaged model performs similarly on the task for which the base models in the average were trained.
Revised: We ﬁnd that this averaged model performs similarly on the task for which the base models in the average were trained.

Original: We also use the model embedding space to extrapolate outside the space of base models and ﬁnd cases in which the model embedding manifold speciﬁes models that perform better on a task than any trained base model.
Revised: We also use the model embedding space to extrapolate outside the space of base models and ﬁnd cases in which the model embedding manifold speciﬁes models that perform better on a task than any trained base model.

Original: We explain later on  how this can be regarded as a form of semi-supervised learning.
Revised: Later on we will explain how this can be regarded as a form of semi-supervised learning.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: We ﬁrst provide the mathematical setup for our algorithmic construction of model embedding spaces.
Revised: We ﬁrst provide the mathematical setup for our algorithmic construction of model embedding spaces.

Original: After reviewing related work, we then provide results of numerical experiments implementing the algorithm and explore clustering, model averaging, and semi-supervised learning on the model embedding space.
Revised: After reviewing related work, we then present results of numerical experiments which implement the algorithm and explore clustering, model averaging and semi-supervised learning on the model embedding space.

Original: We further examine how topological features of the dynamics of RNNs in model embedding spaces are reﬂective of classes of high-level computation processes.
Revised: We further examine how topological features of the dynamics of RNNs in model embedding spaces are reﬂective of classes of high-level computation processes.

Original: Finally, we conclude with a discussion which sketches a broad conceptual framework in which our work resides, and suggests future directions for experimental explorations as well as mathematical foundations.
Revised: Finally, we conclude with a discussion.

Original:  D YNAMICAL M ODEL E MBEDDING 2. M ATHEMATICAL S ETUP
Revised:  D YNAMICAL M ODEL E MBEDDING 2. M ATHEMATICAL S ETUP

Original: In this section, we provide the mathematical setup for construction of model embedding spaces.
Revised: In this section, we provide the mathematical setup for construction of model embedding spaces.

Original: We treat this in the RNN setting, and relegate the CNN setting to Section A of the Appendix.
Revised: We treat this in the RNN setting, and relegate the CNN setting to Section A of the Appendix.

Original: Input: Base models { ( F n , G n ) } Nn =1 , loss functions { ( L n, hidden , L n, output ) } Nn =1 , datasets {D n } Nn =1 , output loss weight λ ≥
Revised: Input: Base models { ( F n , G n ) } Nn =1 , loss functions { ( L n, hidden , L n, output ) } Nn =1 , datasets {D n } Nn =1 , output loss weight λ ≥

Original: Output: Meta-model ( (cid:101) F, (cid:101) G ) , state maps { V n } Nn =1 , model embeddings { θ n } Nn =
Revised: Output: Meta-model ( (cid:101) F, (cid:101) G ) , state maps { V n } Nn =1 , model embeddings { θ n } Nn =

Original: Initialize networks (cid:101) F , (cid:101) G , { V n } Nn =
Revised: Initialize networks (cid:101) F , (cid:101) G , { V n } Nn =

Original: Initialize model embeddings θ n =for τ = 1 , . .
Revised: Initialize model embeddings θ n =for τ = 1 , . .

Original: , τ max do
Revised: , τ max do

Original: Sample base model ( F i , G i ) uniformly at random
Revised: Sample base model ( F i , G i ) uniformly at random

Original: Sample input { x t } from dataset D i
Revised: Sample input { x t } from dataset D i

Original: Compute base model hidden states { h t } and outputs { G i ( h t ) }
Revised: Compute base model hidden states { h t } and outputs { G i ( h t ) }

Original: Compute meta-model hidden states { h θ i ,t } and outputs { (cid:101) G ( h θ i ,t ) }
Revised: Compute meta-model hidden states { h θ i ,t } and outputs { (cid:101) G ( h θ i ,t ) }

Original: Compute mapped meta-model states { V i ( h θ i ,t ) }
Revised: Compute mapped meta-model states { V i ( h θ i ,t ) }

Original: Compute loss (cid:98) L = L i, hidden + λ L i, output
Revised: Compute loss (cid:98) L = L i, hidden + λ L i, output

Original: Update (cid:101) F , (cid:101) G , V i , θ i using ∇ (cid:98) L end for return ( (cid:101) F, (cid:101) G ) , { V n } , { θ n }
Revised: Update (cid:101) F , (cid:101) G , V i , θ i using ∇ (cid:98) L end for return ( (cid:101) F, (cid:101) G ) , { V n } , { θ n }

Original: Notation.
Revised: Notation.

Original: Let an RNN be denoted by F ( x, h ) where x is the input and h is the hidden state.
Revised: Let an RNN be denoted by F ( x, h ) where x is the input and h is the hidden state.

Original: We further denote the hidden-to-output map by G ( h ) .
Revised: We further denote the hidden-to-output map by G ( h ) .

Original: We consider a collection of N RNNs { ( F n , G n ) }
Revised: We consider a collection of N RNNs { ( F n , G n ) }

Original: Nn = we call the base models which may each have distinct dimensions for their hidden states, butwill all have the same dimension for their inputs as well as the same dimension for their output states.
Revised: Nn = we call the base models which may each have distinct dimensions for their hidden states, but all have the same dimension for their inputs as well as the same dimension for their outputs.

Original: A sequence of inputs is notated as { x t } Tt =0 which induces a sequence of hidden states by h t +1 = F ( x t , h t ) where the initial hidden state h 0 is given.
Revised: A sequence of inputs is notated as { x t } Tt =0 which induces a sequence of hidden states by h t +1 = F ( x t , h t ) where the initial hidden state h 0 is given.

Original: A collection of sequences of inputs, possibly each with different maximum lengths T , is denoted by D which we call an input data set .
Revised: A collection of sequences of inputs, possibly each with different maximum lengths T , is denoted by D which we call an input data set .

Original: We suppose that each base model RNN has an associated input data set.
Revised: We suppose that each base model RNN has an associated input data set.

Original: 2. M ETA -M ODELS FOR RNN S
Revised: 2. M ETA -M ODELS FOR RNN S

Original: Given a collection base model RNNs, we would like to construct a meta-model which emulates the behavior of each of the base models.
Revised: Given a collection base model RNNs, we would like to construct a meta-model which emulates the behavior of each of the base models.

Original: In this case, the meta-model is itself an RNN with one additional input θ ∈ R d and a corresponding map (cid:101) F ( θ, x, h ) whose output is the next hidden state.
Revised: In this case, the meta-model is itself an RNN with one additional input θ ∈ R d and a corresponding map (cid:101) F ( θ, x, h ) whose output is the next hidden state.

Original: Given a sequence of input states { x t } Tt =0 , we have a corresponding sequence of output states h θ,t +1 =
Revised: Given a sequence of input states { x t } Tt =0 , we have a corresponding sequence of output states h θ,t +1 =

Original: (cid:101) F ( θ, x t , h θ,t ) starting from an initial hidden state h 0 (which we suppose does not depend on θ ).
Revised: (cid:101) F ( θ, x t , h θ,t ) starting from an initial hidden state h 0 (which we suppose does not depend on θ ).

Original: The meta-model also includes a hidden-to-output map (cid:101) G ( h ) that is independent of θ .
Revised: The meta-model also includes a hidden-to-output map (cid:101) G ( h ) that is independent of θ .

Original: For the meta-model ( (cid:101) F, (cid:101) G ) to emulate a particular base model ( F n , G n ) with respect to its corresponding data set D n , we consider the following criteria: there is some θ n for which
Revised: For the meta-model ( (cid:101) F, (cid:101) G ) to emulate a particular base model ( F n , G n ) with respect to its corresponding data set D n , we consider the following criteria: there is some θ n for which

Original: 1. (cid:101) G ( h θ n ,t )
Revised: 1. (cid:101) G ( h θ n ,t )

Original: ≈ G n ( h t ) for all t > 0 and all input sequences in the data set 2.
Revised: ≈ G n ( h t ) for all t > 0 and all input sequences in the data set 2.

Original: V n ( h θ,t ) ≈ h t for all t > 0 and all input sequences in the data set,
Revised: V n ( h θ,t ) ≈ h t for all t > 0 and all input sequences in the data set,

Original: where V n is a transformation of a meta-model’s hidden activity.
Revised: where V n is a transformation of a meta-model’s hidden activity.

Original: We emphasize that θ n and V n depend on the particular base model under consideration.
Revised: We emphasize that θ n and V n depend on the particular base model under consideration.

Original: The ﬁrst criterion means that at some particular θ n , the outputs of the meta-model RNN dynamics are close to the outputs of the base model RNN dynamics.
Revised: The ﬁrst criterion means that at some particular θ n , the outputs of the meta-model RNN dynamics are close to the outputs of the base model RNN dynamics.

Original: The second criterion means that at the same θ n , there is a time-independent transformation V n (i.e., V n does not depend on t )
Revised: The second criterion means that at the same θ n , there is a time-independent transformation V n (i.e., V n does not depend on t )

Original: such the transformed hidden state dynamics of the meta-model are close to the hidden state dynamics of the base model.
Revised: such the transformed hidden state dynamics of the meta-model are close to the hidden state dynamics of the base model.

Original: See Figure 2 for a visualization.
Revised: See Figure 2 for a visualization.

Original: As depicted in the ﬁgure, it is convenient to regard the meta-model RNN as having input states ( θ n , x ) .
Revised: As depicted in the ﬁgure, it is convenient to regard the meta-model RNN as having input states ( θ n , x ) .

Original: As such, a sequence of inputs { x t } Tt =1 is appended by θ n to become { ( θ n , x t ) } Tt =1 .
Revised: As such, a sequence of inputs { x t } Tt =1 is appended by θ n to become { ( θ n , x t ) } Tt =1 .

Original: The desired properties of the meta-model are enforced in the loss function.
Revised: The desired properties of the meta-model are enforced in the loss function.

Original: Deﬁning the functions
Revised: Deﬁning the functions

Original: [Equation]
Revised: [Equation]

Original: where d is some suitable distance or divergence, we can construct the loss function
Revised: where d is some suitable distance or divergence, we can construct the loss function

Original: [Equation]
Revised: [Equation]

Original: where we average over the choice of sequence { x t } coming from the input data set D n .
Revised: where we average over the choice of sequence { x t } coming from the input data set D n .

Original: Above, λ is a hyperparameter.
Revised: Above, λ is a hyperparameter.

Original: Our aim is to minimize the above over a suitable class of (cid:101) F, (cid:101) G, V n , as well as θ n .
Revised: Our aim is to minimize the above over a suitable class of (cid:101) F, (cid:101) G, V n , as well as θ n ; this can be implemented computationally via the D YNAMO algorithm (see Algorithm 1).

Original: As a side remark, it appears that a suitable alternative choice to equation 2 would be
Revised: As a side remark, it appears that a suitable alternative choice to equation 2 would be

Revised: T

Original: [Equation]
Revised: [Equation]

Original: 22 where here W n is a map from the hidden states of the base model to the hidden states of the meta-model.
Revised: 22 where here W n is a map from the hidden states of the base model to the hidden states of the meta-model.

Original: However, this would be problematic since minimization may pressure W n to be the the zero map (or otherwise have outputs which are small in norm) and accordingly pressure the dynamics of the meta-model to be the trivial (or have small norm).
Revised: However, this would be problematic since minimization may pressure W n to be the the zero map (or otherwise have outputs which are small in norm) and accordingly pressure the dynamics of the meta-model to be the trivial (or have small norm).

Original: As such, we opt to formulate L hidden as it is written in equation 2.
Revised: As such, we opt to formulate L hidden as it is written in equation 2.

Original: Suppose we want the meta-model to be able to emulate an entire collection of base models { ( F n , G n ) }
Revised: Suppose we want the meta-model to be able to emulate an entire collection of base models { [Equation]

Original: Nn =1 .
Revised: =1 .

Original: In particular, the meta-model will attempt to assign to the n th base model a θ n and V n so that the two criteria listed above are satisﬁes for that base model.
Revised: In particular, the meta-model will attempt to assign to the n th base model a θ n and V n so that the two criteria listed above are satisﬁes for that base model.

Original: These desiderata can be implemented by minimizing the loss function
Revised: These desiderata can be implemented by minimizing the loss function

Original: [Equation]
Revised: [Equation]

Original: In some circumstances, we may want to consider base models with distinct dimensions for their output states.
Revised: In some circumstances, we may want to consider base models with distinct dimensions for their output states.

Original: For instance, suppose half of the base models perform a task with outputs in R d 1 and the other half perform a task without outputs in R d 2 .
Revised: For instance, suppose half of the base models perform a task with outputs in R d 1 and the other half perform a task without outputs in R d 2 .

Original: To accommodate for this, we can have two hidden-to-output maps (cid:101) G 1 , (cid:101) G 2 for the meta-model, where the maps have outputs in R d 1 and R drespectively.
Revised: To accommodate for this, we can have two hidden-to-output maps (cid:101) G 1 , (cid:101) G 2 for the meta-model, where the maps have outputs in R d 1 and R drespectively.

Original: The loss function is slightly modiﬁed so that we use (cid:101) G 1 where compare the metamodel to the ﬁrst kind of base model, and (cid:101) G 2 when we compare the meta-model to the second kind of base model.
Revised: The loss function is slightly modiﬁed so that we use (cid:101) G 1 where compare the metamodel to the ﬁrst kind of base model, and (cid:101) G 2 when we compare the meta-model to the second kind of base model.

Original: This construction generalizes to the case where the base models can be divided up into k groups with distinct output dimensions; this would necessitate k hidden-to-output functions (cid:101) G 1 , ..., (cid:101) G k for the meta-model.
Revised: This construction generalizes to the setting where the base models can be divided up into k groups with distinct output dimensions; this would necessitate k hidden-to-output functions (cid:101) G 1 , ..., (cid:101) G k for the meta-model.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There is a substantial body of work on interpreting the computational processes implemented neural networks by studying their intermediate representations.
Revised: There is a substantial body of work on interpreting the computational processes implemented neural networks by studying their intermediate representations.

Original: Such analyses can be performed either on individual models (Simonyan et al., 2013; Zeiler & Fergus, 2014; Lenc & Vedaldi, 2015) or on collections of models (Li et al., 2016;
Revised: Such analyses can be performed either on individual models (Simonyan et al., 2013; Zeiler & Fergus, 2014; Lenc & Vedaldi, 2015) or on collections of models (Li et al., 2016;

Original: Raghu et al., 2017; Morcos et al., 2018; Kornblith et al., 2019).
Revised: Raghu et al., 2017; Morcos et al., 2018; Kornblith et al., 2019).

Original: Prior work in this latter category has focused on pairwise comparisons between models.
Revised: Prior work in this latter category has focused on pairwise comparisons between models.

Original: For example, SVCCA (Raghu et al., 2017) uses canonical correlation analysis to measure the representational similarity between pairs of models.
Revised: For example, SVCCA (Raghu et al., 2017) uses canonical correlation analysis to measure the representational similarity between pairs of models.

Original: While these methods can also be used to derive model representations by embedding the pairwise distance matrix, our approach does not require the Θ( N 2 ) computational cost of comparing all pairs of base models.
Revised: While these methods can also be used to derive model representations by embedding the pairwise distance matrix, our approach does not require the Θ( N 2 ) computational cost of comparing all pairs of base models.

Original: Moreover, D YNAMO yields an executable meta-model that can be called with model embedding vectors other than those corresponding to the base models.
Revised: Moreover, D YNAMO yields an executable meta-model that can be called with model embedding vectors other than those corresponding to the base models.

Original: There is a related body of work in the ﬁeld of computational neuroscience.
Revised: There is a related body of work in the ﬁeld of computational neuroscience.

Original: CCA based techniques and representational geometry are standard for comparing neural networks to the neural activations of animals performing vision tasks (Yamins & DiCarlo, 2016) as well as motor tasks.
Revised: CCA based techniques and representational geometry are standard for comparing neural networks to the neural activations of animals performing vision tasks (Yamins & DiCarlo, 2016) as well as motor tasks.

Original: In an example of the latter, the authors of (Sussillo et al., 2015) used CCA techniques to compare brain recordings to those of neural networks trained to reproduce the reaching behaviors of animals, while the authors of (Maheswaranathan et al., 2019b) used ﬁxed point analyses of RNNs to consider network similarity from a topological point of view.
Revised: In an example of the latter, the authors of (Sussillo et al., 2015) used CCA techniques to compare brain recordings to those of neural networks trained to reproduce the reaching behaviors of animals, while the authors of (Maheswaranathan et al., 2019b) used ﬁxed point analyses of RNNs to consider network similarity from a topological point of view.

Original: D YNAMO can be viewed as a form of knowledge distillation (Hinton et al., 2015), since the outputs of the base models serve as targets in the optimization of the meta-model.
Revised: D YNAMO can be viewed as a form of knowledge distillation (Hinton et al., 2015), since the outputs of the base models serve as targets in the optimization of the meta-model.

Original: However, unlike typical instances of knowledge distillation involving an ensemble of teacher networks (Hinton et al., 2015;
Revised: However, unlike typical instances of knowledge distillation involving an ensemble of teacher networks (Hinton et al., 2015;

Original: Fukuda et al., 2017) where individual model predictions are averaged to provide more accurate target labels for the student, our approach instead aims to preserve the dynamics and outputs of each individual base model.
Revised: Fukuda et al., 2017) where individual model predictions are averaged to provide more accurate target labels for the student, our approach instead aims to preserve the dynamics and outputs of each individual base model.

Original: FitNets (Romero, 2015) employ a form a knowledge distillation using maps between hidden representations to guide learning; this is similar to our use of hidden state maps in training the meta-model.
Revised: FitNets (Romero, 2015) employ a form a knowledge distillation using maps between hidden representations to guide learning; this is similar to our use of hidden state maps in training the meta-model.

Original: Our treatment of the model embedding vectors { θ n } as learnable parameters is similar to the approach used in Generative Latent Optimization (Bojanowski et al., 2018), which jointly optimizes the image generator network and the latent vectors corresponding to each image in the training set.
Revised: Our treatment of the model embedding vectors { θ n } as learnable parameters is similar to the approach used in Generative Latent Optimization (Bojanowski et al., 2018), which jointly optimizes the image generator network and the latent vectors corresponding to each image in the training set.

Original: Bojanowski et al.
Revised: Bojanowski et al.

Original: ﬁnd that the principal components of the image representation space found by GLO are semantically meaningful.
Revised: ﬁnd that the principal components of the image representation space found by GLO are semantically meaningful.

Original: We likewise ﬁnd that the principal components of the model embeddings found by D YNAMO are discriminative between subsets of models.
Revised: We likewise ﬁnd that the principal components of the model embeddings found by D YNAMO are discriminative between subsets of models.

Original: Unlike methods like hypernetworks (Ha et al., 2017) and LEO (Rusu et al., 2019) that use a model to generate parameters for a separate network, our approach does not attempt to reproduce the parameters of the base models in the collection.
Revised: Unlike methods such as hypernetworks (Ha et al., 2017) and LEO (Rusu et al., 2019) that use a model to generate parameters for a separate network, our approach does not attempt to reproduce the parameters of the base models in the collection.

Original: Instead, the meta-model aims to reproduce only the hidden states and outputs of a base model when conditioned on the corresponding embedding vector.
Revised: Instead, the meta-model aims to reproduce only the hidden states and outputs of a base model when conditioned on the corresponding embedding vector.

Original: The core focus of this work also differs from that of the meta-learning literature (Santoro et al., 2016; Ravi & Larochelle, 2017; Finn et al., 2017; Munkhdalai & Yu, 2017), which primarily concerns the problem of few-shot adaptation when presented with data from a new task.
Revised: The core focus of this work also differs from that of the meta-learning literature (Santoro et al., 2016; Ravi & Larochelle, 2017; Finn et al., 2017; Munkhdalai & Yu, 2017), which primarily concerns the problem of few-shot adaptation when presented with data from a new task.

Original: Our empirical study centers on the post-hoc analysis of a given collection of models, which may or may not have been trained on different tasks.
Revised: Our empirical study centers on the post-hoc analysis of a given collection of models, which may or may not have been trained on different tasks.

Original: However, we remark that our exploration of optimization in lowdimensional model embedding space is related to LEO (Rusu et al., 2019), where a compressed model representation is leveraged for efﬁcient meta-learning.
Revised: However, we remark that our exploration of optimization in lowdimensional model embedding space is related to LEO (Rusu et al., 2019), where a compressed model representation is leveraged for efﬁcient meta-learning.

Original:  E MPIRICAL R ESULTS
Revised:  E MPIRICAL R ESULTS

Original: In this section, we describe the results of our empirical study of meta-models trained using D YNAMO on collections of RNNs trained on NLP tasks, and collections of CNNs trained for image classiﬁcation.
Revised: In this section, we describe the results of our empirical study of meta-models trained using D YNAMO on collections of RNNs trained on NLP tasks, and collections of CNNs trained for image classiﬁcation.

Original: For RNN base models, we parameterize the meta-model as a GRU where the model embedding vector θ is presented as an additional input in each time step.
Revised: For RNN base models, we parameterize the meta-model as a GRU where the model embedding vector θ is presented as an additional input in each time step.

Original: For CNN base models, we use a ResNet meta-model where θ is an additional input for each ResNet block.
Revised: For CNN base models, we use a ResNet meta-model where θ is an additional input for each ResNet block.

Original: In all our experiments, we hold out half the available training data for use as unlabeled data for training the meta-model; the base models were trained on the remaining training data (or a fraction thereof).
Revised: In all our experiments, we hold out half the available training data for use as unlabeled data for training the meta-model; the base models were trained on the remaining training data (or a fraction thereof).

Original: 1 By default, we set the output loss hyperparameter λ to 1 .
Revised: 1 By default, we set the output loss hyperparameter λ to 1 .

Original: We defer further details on model architectures and training to the Appendix.
Revised: We defer further details on model architectures and training to the Appendix.

Original: 4. V ISUALIZING M ODEL S IMILARITY IN E MBEDDING S PACE
Revised: 4. V ISUALIZING M ODEL S IMILARITY IN E MBEDDING S PACE

Original: The base model embeddings { θ n } Nn =1 can be used for cluster analysis to evaluate the similarity structure of a collection of models.
Revised: The base model embeddings { θ n } Nn =1 can be used for cluster analysis to evaluate the similarity structure of a collection of models.

Original: We illustrate this use-case of D YNAMO via a series of example applications on NLP and vision tasks.
Revised: We illustrate this use case of D YNAMO via a series of example applications on NLP and vision tasks.

Original: Figure 3 shows model embeddings learned by D YNAMO on collections of RNNs.
Revised: Figure 3 shows model embeddings learned by D YNAMO on collections of RNNs.

Original: In these plots, we observe a clear separation of these networks according to the size of the available training data, the RNN model architecture used, and the speciﬁc NLP task used for training.
Revised: In these plots, we observe a clear separation of these networks according to the size of the available training data, the RNN model architecture used, and the speciﬁc NLP task used for training.

Original: By computing the eigenvalues of the covariance matrix corresponding to the N model embeddings, we obtain a measure of the intrinsic dimensionality of the corresponding collections of models.
Revised: By computing the eigenvalues of the covariance matrix corresponding to the N model embeddings, we obtain a measure of the intrinsic dimensionality of the corresponding collections of models.

Original: In Figure 3,we ﬁndthat 2 to 6 components are sufﬁcient to explain 95% of the variancein the modelembeddings.
Revised: In Figure 3, ﬁnd 2 to 6 components are sufﬁcient to explain 95% of the variance the model

Original: By setting the output loss hyperparameter λ in equation 4, we can adjust the degree of emphasis placed on reproducing the hidden dynamics of the base networks vs. their outputs.
Revised: By tuning the output loss hyperparameter λ in equation 4, we can adjust the degree of emphasis placed on reproducing the hidden dynamics of the base networks versus their outputs.

Original: We demonstrate this effect in Figure 4: with λ = 1 , we observe clear separation of ResNet-34 models trained on CIFAR-100 with different data augmentation policies (“weak”, with shifts and horizontal ﬂips vs. “strong”, with RandAugment (Cubuk et al., 2020)), and with different training set sizes.
Revised: We demonstrate this effect in Figure 4: with λ = 1 , we observe clear separation of ResNet-34 models trained on CIFAR-100 with different data augmentation policies (“weak”, with shifts and horizontal ﬂips vs. “strong”, with RandAugment (Cubuk et al., 2020)) and with different training set sizes.

Original: In contrast, with λ = 0 , we observe a weak clustering effect with differing data augmentation, and no detectable separation with different training set sizes.
Revised: In contrast, with λ = 0 , we observe a weak clustering effect with differing data augmentation and no detectable separation with different training set sizes.

Original: We infer that the change in data augmentation policy results in a larger difference in the learned feature representations than the change in training set size.
Revised: We infer that the change in data augmentation policy results in a larger difference in the learned feature representations than the change in training set size.

Original: In Appendix B, we illustrate the effect of setting λ = 0 for RNN models.
Revised: In Appendix B, we illustrate the effect of setting λ = 0 for RNN models.

Original: We additionally compare the embeddings obtained using D YNAMO to those derived from
Revised: We additionally compare the embeddings obtained using D YNAMO to those derived from

Original: SVCCA (Raghu et al., 2017), a pairwise comparison technique that aligns the representations produced by a pair of networks using canonical correlation analysis (CCA).
Revised: SVCCA (Raghu et al., 2017), a pairwise comparison technique that aligns the representations produced by a pair of networks using canonical correlation analysis (CCA).

Original: In Figure 5, we plot the 2D embeddings obtained using multidimensional scaling (MDS) on the pairwise distance matrix computed using SVCCA (modiﬁed to output L 2 distances instead of correlations).
Revised: In Figure 5, we plot the 2D embeddings obtained using multidimensional scaling (MDS) on the pairwise distance matrix computed using SVCCA (modiﬁed to output L 2 distances instead of correlations).

Original: Unlike the principal components of the D YNAMO model embeddings plotted in Figure 3, the MDS coordinates are not semantically interpretable.
Revised: Unlike the principal components of the D YNAMO model embeddings plotted in Figure 3, the MDS coordinates are not semantically interpretable.

Original: Additionally, the cluster structure of the collection of GRUs trained with varying training set sizes is less apparent in this representation.
Revised: Additionally, the cluster structure of the collection of GRUs trained with varying training set sizes is less apparent in this representation.

Original: Lastly, we note that D YNAMO allows for ﬂexibility in deﬁning the metric used to compare the hidden states and outputs of the base models with those of the meta-model.
Revised: Lastly, we note that D YNAMO allows for ﬂexibility in deﬁning the metric used to compare the hidden states and outputs of the base models with those of the meta-model.

Original: In Appendix B.3, we demonstrate the beneﬁt of using the L 1 distance for clustering CNN representations.
Revised: In Appendix B.3, we demonstrate the beneﬁt of using the L 1 distance for clustering CNN representations.

Original: 4. E XTRAPOLATION B EYOND B ASE M ODEL E MBEDDINGS
Revised: 4. E XTRAPOLATION B EYOND B ASE M ODEL E MBEDDINGS

Original: We study the model embedding space corresponding to a trained meta-model by conditioning it on model embedding vectors θ other than those assigned to the set of base models.
Revised: We study the model embedding space corresponding to a trained meta-model by conditioning it on model embedding vectors θ other than those assigned to the set of base models.

Original: Figure 6 visualizes the landscape of test accuracies for two meta-models: (i) a meta-model for 10 GRUs trained with50% of the IMDB training data, and 10 GRUs trained with 25% of the data, and (ii) a meta-model for 10 IMDB sentiment GRUs and 10 AG News classiﬁcation GRUs.
Revised: Figure 6 visualizes the landscape of test accuracies for two meta-models: (i) a meta-model for 10 GRUs trained with

Original: We note two particularly salient properties of these plots.
Revised: We note two particularly salient properties of these plots.

Original: First, the test accuracy varies smoothly when interpolating θ between pairs of base model embeddings—we would in general not observe this property when interpolating the parameters of the base GRUs directly, since they were trained with different random initializations and orderings of the training examples.
Revised: First, the test accuracy varies smoothly when interpolating θ between pairs of base model embeddings—we would in general not observe this property when interpolating the parameters of the base GRUs directly, since they were trained with different random initializations and orderings of the training examples.

Original: Second, we observe that the embedding vector that realizes the highest test accuracy lies outside the convex hull of the base model embeddings.
Revised: Second, we observe that the embedding vector that realizes the highest test accuracy lies outside the convex hull of the base model embeddings.

Original: This is perhaps surprising since typical training and inference protocols involve the use of convex combinations of various objects: for instance, averaging of predictions in model ensembles, and averaging of model parameters during training (e.g., using exponential moving averages, or with Stochastic Weight Averaging (Izmailov et al., 2018)).
Revised: This is perhaps surprising since typical training and inference protocols involve the use of convex combinations of various objects: for instance, averaging of predictions in model ensembles, and averaging of model parameters during training (e.g., using exponential moving averages, or with Stochastic Weight Averaging (Izmailov et al., 2018)).

Original: This extrapolatory phenomenon suggests that D YNAMO is able to derive a low-dimensional manifold of models that generalizes beyond the behavior of the base models used for training the meta-model.
Revised: This extrapolatory phenomenon suggests that D YNAMO is able to derive a low-dimensional manifold of models that generalizes beyond the behavior of the base models used for training the meta-model.

Original: 4. S EMI -S UPERVISED L EARNING IN M ODEL E MBEDDING S PACE
Revised: [Figure]

Original: The existence of model embeddings that improve on the accuracy of the base models suggests a natural semi-supervised learning  procedure involving a trained meta-model.
Revised: The existence of model embeddings that improve on the accuracy of the base models suggests a natural semi-supervised learning (SSL) procedure involving a trained meta-model.

Original: In particular, we minimize the loss incurred by the meta-model on a small set of additional labeled examples by optimizing the value of θ .
Revised: In particular, we minimize the loss incurred by the meta-model on a small set of additional labeled examples by optimizing the value of θ .

Original: This is done by backpropagating gradients through the meta-model, with the meta-model parameters held ﬁxed.
Revised: This is done by backpropagating gradients through the meta-model, with the meta-model parameters held ﬁxed.

Original: Figure 7 shows the result of this procedure on an IMDB sentiment meta-model (previously depicted in Figure 6) with a set of additional labeled examples (disjoint from the test set) of size equal to 1% of the full training set.
Revised: Figure 7 shows the result of this procedure on an IMDB sentiment meta-model (previously depicted in Figure 6) with a set of additional labeled examples (disjoint from the test set) of size equal to 1% of the full training set.

Original: This procedure successfully ﬁnds a θ that improves on the test accuracy of the best base model by 6% ( 86 . 4% vs. 80 .
Revised: This procedure successfully ﬁnds a θ that improves on the test accuracy of the best base model by 6% ( 86 . 4% vs. 80 .

Original: We observe that this SSL procedure achieves lower accuracy when we train the meta-model using fewer base models.
Revised: We observe that this SSL procedure achieves lower accuracy when we train the meta-model using fewer base models.

Original: In particular, a metamodel only on the 10 GRUs trained with 50% of the training data yields a test accuracy of 85 .
Revised: In particular, a metamodel only on the 10 GRUs trained with 50% of the training data yields a test accuracy of 85 .

Original: 4% , and a meta-model on only a single GRU out of these 10 yields 81 .
Revised: 4% , and a meta-model on only a single GRU out of these 10 yields 81 .

Original: This result suggests that a diversity of base models helps improve the accuracy achievable by the meta-model.
Revised: This result suggests that a diversity of base models helps improve the accuracy achievable by the meta-model.

Original:  D YNAMICS OF M ETA -M ODELS FOR RNN S
Revised:  D YNAMICS OF M ETA -M ODELS FOR RNN S

Original: In this section we perform an analysis of the dynamical features of the dynamics generated by the meta-models trained on base models that perform the sentiment classiﬁcation task.
Revised: In this section we perform an analysis of the dynamical features of the dynamics generated by the meta-models trained on base models that perform the sentiment classiﬁcation task.

Original: The sentiment classiﬁcation task has a well-understood dynamical structure (Sussillo & Barak, 2013; Maheswaranathan et al., 2019a;b; Aitken et al., 2020) that we can use as a basis for understanding the behavior of a corresponding meta-model.
Revised: The sentiment classiﬁcation task has a well-understood dynamical structure (Sussillo & Barak, 2013; Maheswaranathan et al., 2019a;b; Aitken et al., 2020) that we can use as a basis for understanding the behavior of a corresponding meta-model.

Original: To a ﬁrst approximation, the sentiment analysis task can be solved by a simple integrator that accumulates sentiment corresponding to each word (positive words such as ‘good’ or ‘fantastic’ adding positive sentiment, and negative words such as ‘bad’ or ‘terrible’ adding negative sentiment).
Revised: To a ﬁrst approximation, the sentiment analysis task can be solved by a simple integrator that accumulates sentiment corresponding to each word (positive words such as ‘good’ or ‘fantastic’ adding positive sentiment, and negative words such as ‘bad’ or ‘terrible’ adding negative sentiment).

Original: It has been shown that simple sentiment analysis models approximate this integrator by constructing a line attractor in the space of hidden states.
Revised: It has been shown that simple sentiment analysis models approximate this integrator by constructing a line attractor in the space of hidden states.

Original: For instance, for the zero input x ∗ =
Revised: For instance, for the zero input x ∗ =

Original: (cid:126) 

Revised: (cid:126)

Original: , it has been observed that the dynamical system generated by the map
Revised: , it has been observed that the dynamical system generated by the map

Original: F x ∗ ( h ) =
Revised: F x ∗ ( h ) =

Original: F ( x ∗ , h ) has a tubular region with very little movement, in the sense
Revised: F ( x ∗ , h ) has a tubular region with very little movement, in the sense

Original: that (cid:107) F x ∗ ( h ) − h
Revised: that (cid:107) F x ∗ ( h ) − h

Original: (cid:107)is very small for hidden states h in this region.
Revised: (cid:107)is very small for hidden states h in this region.

Original: To investigate the dynamical behavior of the meta-model space, we trained a meta-model on a set of 20 base models  trained on the IMDB sentiment analysis task, 10  trained with 50% of the available training data and the remaining 10  on 100% of the training data.
Revised: To investigate the dynamical behavior of the meta-model space, we trained a meta-model on a set of 20 base models which were themselves trained on the IMDB sentiment analysis task. Of these 20 base models, 10 were trained with 50% of the available training data and the remaining 10 were trained on 100% of the training data.

Original: These points  clustered in the model embedding space according to the amount of training data.
Revised: The θ points corresponding to these base models clustered in the model embedding space according to the amount of training data.

Original: In Figure 8 we perform a ﬁxed-point analysis of several models corresponding to points of interest in the model embedding space.
Revised: In Figure 8 we perform a ﬁxed-point analysis of several models corresponding to points of interest in the model embedding space.

Original: The ﬁxed-point analysis was run according to the procedure described in (Golub & Sussillo, 2018).
Revised: The ﬁxed-point analysis was run according to the procedure described in (Golub & Sussillo, 2018).

Original: First we selected a set of candidate hidden states h j by running the model on a typical batch of inputs.
Revised: First we selected a set of candidate hidden states h j by running the model on a typical batch of inputs.

Original: For each hidden state h j obtained in this way, we used gradient descent on the loss (cid:107) F ( x ∗ , h ) − h
Revised: For each hidden state h j obtained in this way, we used gradient descent on the loss (cid:107) F ( x ∗ , h ) − h

Original: (cid:107)to ﬁnd the nearest approximate ﬁxed point.
Revised: (cid:107)to ﬁnd the nearest approximate ﬁxed point.

Original: An interesting ﬁnding is that the meta-model found line attractor structures that were very geometrically similar for models within a cluster.
Revised: An interesting ﬁnding is that the meta-model found line attractor structures that were very geometrically similar for models within a cluster.

Original: An interpretation of this result pertaining to topological conjugacy in dynamical systems theory is discussed in Section C of the Supplementary Materials.
Revised: An interpretation of this result pertaining to topological conjugacy in dynamical systems theory is discussed in Section C of the Supplementary Materials.

Original: Moreover we ﬁnd that the meta-model ﬁnds a continuous interpolation between line attractors that are relatively fat and thin (corresponding to models trained on 50% of the data), and models that are elongated (corresponding to models trained on 100% of the data).
Revised: Moreover, we ﬁnd that the meta-model ﬁnds a continuous interpolation between line attractors that are relatively fat and thin (corresponding to models trained on 50% of the data), and models that are elongated (corresponding to models trained on 100% of the data).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have introduced the algorithm D YNAMO , which maps a set of neural network base models to a low dimensional feature space Our results show that the model embeddings provided by D YNAMO capture relevant computational features of the base models.
Revised: We have introduced the algorithm D YNAMO , which maps a set of neural network base models to a low dimensional feature space. Our results show that the model embeddings provided by D YNAMO capture relevant computational features of the base models.

Original: Moreover, the space of models described by D YNAMO seems to describe a  smoothspace of models so that model averaging can be performed, and even model extrapolation to reach new models with better performance than any base model.
Revised: Moreover, the model embedding spaces produced by D YNAMO are a sufﬁciently smooth that model averaging can be performed, and even model extrapolation to reach new models with better performance than any base model.

Original: Indeed, in the case that the base models are trained on the sentiment analysis task, the model embedding space describes a space of line attractors which vary smoothly in the parameter θ .
Revised: Indeed, in our experiments where the base models were trained on the sentiment analysis task, the model embedding space describes a space of line attractors which vary smoothly in the parameter θ .

Original: It is possible to apply D YNAMO  to other networks that have a dynamical structure, for example using the depth in a convolutional neural networks as an artiﬁcial time variable.
Revised: We have demonstrated that D YNAMO can be applied broadly to neural networks that have a dynamical structure; for example, we used the demarcation of layers of convolutional neural networks as a proxy for a dynamical time variable.

Original: This also suggests possible scientiﬁc applications of D YNAMO to dynamical systems arising in nature.
Revised: This also suggests possible scientiﬁc applications of D YNAMO to dynamical systems arising in nature.

Original: A  limitation of D YNAMO is the need for all base models to have the same input structure and access to the full hidden state information of the base models.
Revised: A present limitation of D YNAMO is the need for all base models to have the same input structure.

Original: For example, one cannot yet apply D YNAMO to compare language models trained with different encodings (character-based vs. word-based, for example).
Revised: For example, one cannot presently utilize D YNAMO to compare language models trained with different encodings (character-based vs. word-based, for example).

Original: A M ETA - MODELS FOR CNN S
Revised: A M ETA - MODELS FOR CNN S

Original: Our setup in Section 2.2 above can be readily adapted to CNNs, or feedforward neural networks more broadly.
Revised: Our setup in Section 2.2 above can be readily adapted to CNNs, or feedforward neural networks more broadly.

Original: For instance, in the case of ResNets, there is a single input x 0 followed by a sequence of blocks which operate on different numbers of channels.
Revised: For instance, in the case of ResNets, there is a single input x 0 followed by a sequence of blocks which operate on different numbers of channels.

Original: We let the meta-model likewise be a
Revised: We let the meta-model likewise be a

Original: ResNet with the same block structure.
Revised: ResNet with the same block structure.

Original: Moreover, we consider the output b t of the t th block in place of the G n ( h t ) ’s in equation 1, and take the hidden states between the t and t +1 blocks to be the h t ’s in equation 2.
Revised: Moreover, we consider the output b t of the t th block in place of the G n ( h t ) ’s in equation 1, and take the hidden states between the t and t +1 blocks to be the h t ’s in equation 2.

Original: Since each block of a ResNet is distinct, we consider a family of maps V n,t which depend on the base model n and the block layer t .
Revised: Since each block of a ResNet is distinct, we consider a family of maps V n,t which depend on the base model n and the block layer t .

Original: Suppose B is the total number of blocks.
Revised: Suppose B is the total number of blocks.

Original: Then we can more explicitly write
Revised: Then we can more explicitly write

Original: [Equation]
Revised: [Equation]

Original: Then the total loss function accounting for all N of the ResNet base models is
Revised: Then the total loss function accounting for all N of the ResNet base models is

Original: [Equation]
Revised: [Equation]

Original: where we note that in the expectation value we are only sampling over x 0 ’s in D n since x 0 ’s are the only form of input data.
Revised: where we note that in the expectation value we are only sampling over x 0 ’s in D n since x 0 ’s are the only form of input data.

Original: B A DDITIONAL E XPERIMENTAL D ETAILS
Revised: B A DDITIONAL E XPERIMENTAL D ETAILS

Original: In this section, we provide further details on our experiments as well as additional empirical results.
Revised: In this section, we provide further details on our experiments as well as additional empirical results.

Original: B. M ETA -M ODEL A RCHITECTURES
Revised: B. M ETA -M ODEL A RCHITECTURES

Original: RNN architecture.
Revised: RNN architecture.

Original: We parameterize the meta-model for RNNs as a GRU that takes the model embedding vector as an additional input at each time step.
Revised: We parameterize the meta-model for RNNs as a GRU that takes the model embedding vector as an additional input at each time step.

Original: Speciﬁcally, the meta-model GRU takes as input the vector [ x t ; θ ] at each time step, where x t is an input token embedding and [ · ; · ] denotes concatenation.
Revised: Speciﬁcally, the meta-model GRU takes as input the vector [ x t ; θ ] at each time step, where x t is an input token embedding and [ · ; · ] denotes concatenation.

Original: The model embedding vector θ therefore serves as a time-independent bias for the meta-model.
Revised: The model embedding vector θ therefore serves as a time-independent bias for the meta-model.

Original: CNN architecture.
Revised: CNN architecture.

Original: We parameterize the meta-model for CNNs with a modiﬁed ResNet architecture.
Revised: We parameterize the meta-model for CNNs with a modiﬁed ResNet architecture.

Original: In each residual block, we use a linear transformation W to map the model embedding vector θ to the corresponding channel dimension of convolutional layer.
Revised: In each residual block, we use a linear transformation W to map the model embedding vector θ to the corresponding channel dimension of convolutional layer.

Original: We then add the vector Wθ to the channels at each spatial location of the feature map.
Revised: We then add the vector Wθ to the channels at each spatial location of the feature map.

Original: This design emulates the approach used in our parametrization of the RNN meta-model, with the model embedding θ serving as a bias term in each residual block.
Revised: This design emulates the approach used in our parametrization of the RNN meta-model, with the model embedding θ serving as a bias term in each residual block.

Original: We reuse the weight matrix W for all residual blocks with the same channel dimension.
Revised: We reuse the weight matrix W for all residual blocks with the same channel dimension.

Original: Algorithm Model Residual Block
Revised: Algorithm 2 Meta-Model Residual Block

Original: Input: features x , model embedding vector θ z = BatchNorm
Revised: Input: features x , model embedding vector θ z = BatchNorm

Original: ( Conv ( x )) z = ReLU ( z + Wθ ) z = BatchNorm ( Conv ( x ))
Revised: ( Conv ( x )) z = ReLU ( z + Wθ ) z = BatchNorm ( Conv ( x ))

Original: z = ReLU ( x + z ) return z
Revised: z = ReLU ( x + z ) return z

Original: The standard ResNet architecture consists of a sequence of four layers, with each layer consisting of a sequence of residual blocks.
Revised: The standard ResNet architecture consists of a sequence of four layers, with each layer consisting of a sequence of residual blocks.

Original: To compute the hidden state loss L hidden in D YNAMO , we compute distances between the output representations of each of these four layers, averaging over the number of channels and spatial locations in each set of features.
Revised: To compute the hidden state loss L hidden in D YNAMO , we compute distances between the output representations of each of these four layers, averaging over the number of channels and spatial locations in each set of features.

Original: B. T RAINING D ETAILS
Revised: B. T RAINING D ETAILS

Original: Table 1 lists the hyperparameters used for training our RNN base models and meta-models, and
Revised: Table 1 lists the hyperparameters used for training our RNN base models and meta-models, and

Original: Table 2 lists the hyperparameters used for our CNN base models and meta-models.
Revised: Table 2 lists the hyperparameters used for our CNN base models and meta-models.

Original: By default, we use a model embedding dimension of 16 .
Revised: By default, we use a model embedding dimension of 16 .

Original: In the case of visualizing the training trajectories of IMDB sentiment GRUs (Figure 3, second column), we use a model embedding dimension of 32 due to the relatively larger number of base models.
Revised: In the case of visualizing the training trajectories of IMDB sentiment GRUs (Figure 3, second column), we use a model embedding dimension of 32 due to the relatively larger number of base models.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B. S UPPLEMENTARY E MPIRICAL R ESULTS
Revised: B. S UPPLEMENTARY E MPIRICAL R ESULTS

Original: Effect of output loss weight.
Revised: Effect of output loss weight.

Original: Figure 9 shows the effect of setting the output loss weight λ = 0 for meta-models on RNNs.
Revised: Figure 9 shows the effect of setting the output loss weight λ = 0 for meta-models on RNNs.

Original: These plots illustrate that model clustering can be performed on the basis of comparing hidden state dynamics alone.
Revised: These plots illustrate that model clustering can be performed on the basis of comparing hidden state dynamics alone.

Original: We note that the change in the λ hyperparameter results in qualitative changes in the resulting clustering.
Revised: We note that the change in the λ hyperparameter results in qualitative changes in the resulting clustering.

Original: In particular, the model embeddings for GRUs trained on AG News classiﬁcation (rightmost column of Figure 9) are much more tightly coupled relative to the GRUs trained on the IMDB dataset when λ = 0 .
Revised: In particular, the model embeddings for GRUs trained on AG News classiﬁcation (rightmost column of Figure 9) are much more tightly coupled relative to the GRUs trained on the IMDB dataset when λ = 0 .

Original: This indicates that the dynamics implemented by the AG News GRUs are much more similar than those implemented by the IMDB sentiment GRUs.
Revised: This indicates that the dynamics implemented by the AG News GRUs are much more similar than those implemented by the IMDB sentiment GRUs.

Original: Clustering with other loss functions.
Revised: Clustering with other loss functions.

Original: As noted in Sec.
Revised: As noted in Sec.

Original: 4.1, the loss functions used to compare hidden states and outputs can be easily changed to better match the characteristics of the base models under consideration.
Revised: 4.1, the loss functions used to compare hidden states and outputs can be easily changed to better match the characteristics of the base models under consideration.

Original: We demonstrate the beneﬁt of this additional ﬂexibility by replacing the Ldistance in L hidden (equation 6) with the L 1 distance for comparing the intermediate representations of ResNets.
Revised: We demonstrate the beneﬁt of this additional ﬂexibility by replacing the Ldistance in L hidden (equation 6) with the L 1 distance for comparing the intermediate representations of ResNets.

Original: This choice is motivated by the observation that the use of the ReLU nonlinearity results in sparse representations, which suggests the use of the L 1 metric.
Revised: This choice is motivated by the observation that the use of the ReLU nonlinearity results in sparse representations, which suggests the use of the L 1 metric.

Original: Figure 10 shows a comparison between these two distance functions in the case of ResNet-34 models trained on CIFAR-100 with “weak” data augmentation (random shifts and horizontal ﬂips) and with “strong” data augmentation (RandAugment).
Revised: Figure 10 shows a comparison between these two distance functions in the case of ResNet-34 models trained on CIFAR-100 with “weak” data augmentation (random shifts and horizontal ﬂips) and with “strong” data augmentation (RandAugment).

Original: Here, we parameterized the meta-model with a ResNet-50 architecture.
Revised: Here, we parameterized the meta-model with a ResNet-50 architecture.

Original: The use of the L 1 distance results in a clearer separation between the two sets of models.
Revised: The use of the L 1 distance results in a clearer separation between the two sets of models.

Original: This is reﬂected qualitatively in the distribution of the base model embeddings in model embedding space, and quantitatively in the relative scale of the variance captured by the ﬁrst principal component.
Revised: This is reﬂected qualitatively in the distribution of the base model embeddings in model embedding space, and quantitatively in the relative scale of the variance captured by the ﬁrst principal component.

Original: B. F URTHER INVESTIGATION OF THE DYNAMICS OF SENTIMENT ANALYSIS
Revised: B. F URTHER INVESTIGATION OF THE DYNAMICS OF SENTIMENT ANALYSIS

Revised: Recall that our trained RNN’s implemented sentiment analysis via line attractor dynamics, in which inputted words kicked the hidden state in the ‘positive’ or ‘negative’ direction along the line attractor according to the valence of the word (i.e. how positive or negative the word is).

Original: Figure 11 investigates howthe word valences  change as a function of the parameter in model embedding space.
Revised: Figure 11 investigates how valences assigned to words change as we scan across model embedding space.

Original: Given a model corresponding to a value θ , we ﬁrst ﬁnd a ﬁxed point h ∗ with neutral readout G ( h ∗ ) ≈ 0 .
Revised: Given an RNN corresponding to a value θ , we ﬁrst ﬁnd a ﬁxed point h ∗ with neutral readout G ( h ∗ ) ≈ 0 .

Original: For this ﬁxed point, we compute G ( F ( x, h ∗ )) for a variety of word inputs x .
Revised: For this ﬁxed point, we compute G ( F ( x, h ∗ )) for a variety of word inputs x .

Original: To produce a “score” for the model, we compute
Revised: To produce a “score” for the model, we compute

Original: [Equation]
Revised: [Equation]

Original: where the set of good words W good , negative words W negative , and neutral words W neutral are listed in Table 3.
Revised: where the set of positive words W positive , negative words W negative , and neutral words W neutral are listed in Table 3.

Revised: Note that here we are not analyzing context effects, for instance how the string ‘not terrible’ would be rendered into a net-positive valence.

Original: [Table]
Revised: [Table]

Revised: : Lists of good, negative, and neutral words selected to assess the “word valence” quality of a model.

Original: C M ETA - MODELS AND TOPOLOGICAL CONJUGACY
Revised: C M ETA - MODELS AND TOPOLOGICAL CONJUGACY

Original: In this section we describe the notion of topological conjugacy, its relationship to the loss function
Revised: In this section we describe the notion of topological conjugacy, its relationship to the loss function

Original: L hidden , and  speculation as to the interpretation of the results from Section 5.
Revised: L hidden , and provide speculation as to the interpretation of the results from Section 5.

Original: A topological conjugacy (Katok & Hasselblatt, 1997) between two dynamical systems deﬁned by maps F : X → X and E : Y → Y is a homeomorphism Φ : X → Y satisfying
Revised: A topological conjugacy (Katok & Hasselblatt, 1997) between two dynamical systems deﬁned by maps F : X → X and E : Y → Y is a homeomorphism Φ : X → Y satisfying

Original: [Equation]
Revised: [Equation]

Revised: Note that the dynamical systems F and E can have distinct domains.

Original: The signiﬁcance of this relationship  is that dynamics obtained from iterated applications of the map F and E are related to each other by the formula
Revised: The signiﬁcance of the relationship in equation 9 is that dynamics obtained from iterated applications of the map F and E are related to each other by the formula

Original: [Equation]
Revised: [Equation]

Original: Thus, if F and E are topologically conjugate, then their iterates are also topologically conjugate and this means that the dynamics are related by a change of variables.
Revised: Thus, if F and E are topologically conjugate, then their iterates are also topologically conjugate and this means that the dynamics are related by a change of variables.

Original: Notice that topological conjugacy is an equivalence relation; as such the transitive property tells us that if F ∼ E and E ∼ D then
Revised: Notice that topological conjugacy is an equivalence relation; as such the transitive property tells us that if F ∼ E and E ∼ D then

Original: F ∼ D .
Revised: F ∼ D .

Original: The notion of topological conjugacy is an important motivation for deﬁning the loss function L hidden , which we recall is given by
Revised: The notion of topological conjugacy is an important motivation for deﬁning the loss function L hidden , which we recall is given by

Original: [Equation]
Revised: [Equation]

Original: where  h t are dynamics obtained from a base model F ,  h θ t are dynamics obtained from the metamodel (cid:101) F with parameter θ , and V is the map from the meta-model hidden states to the base model hidden states.
Revised: where the hidden states h t are dynamics obtained from a base model F , the hidden states h θ t are dynamics obtained from the meta-model (cid:101) F with parameter θ , and V is the map from the meta-model hidden states to the base model hidden states.

Original: One way of obtaining zero loss is to ﬁnd a topological conjugacy with map V between the meta-model (cid:101) F (at ﬁxed θ ) and the base model F , meaning a relationship of the form
Revised: One way of obtaining zero loss is to ﬁnd a topological conjugacy with map V between the meta-model (cid:101) F (at ﬁxed θ ) and the base model F , meaning a relationship of the form

Original: [Equation]
Revised: [Equation]

Original: It is convenient to deﬁne the notation F x ( h ) :
Revised: It is convenient to deﬁne the notation F x ( h ) :

Original: = F ( x, h ) and (cid:101) F θ,x ( h ) :
Revised: = F ( x, h ) and (cid:101) F θ,x ( h ) :

Original: = (cid:101) F ( θ, x, h ) .
Revised: = (cid:101) F ( θ, x, h ) .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: with h θ, 0 =
Revised: with h θ, 0 =

Original: V − 1 h 0 , so that L hidden [ F , θ, V ] = 0 would hold.
Revised: V − 1 h 0 , so that L hidden [ F , θ, V ] = 0 would hold.

Original: As depicted in Figure 2, to obtain zero loss it would actually sufﬁce to ﬁnd a weaker relationship of the form
Revised: As depicted in Figure 2, to obtain zero loss it would actually sufﬁce to ﬁnd a weaker relationship of the form

Original: [Equation]
Revised: [Equation]

Original: The difference here is that V need not be invertible.
Revised: The difference here is that V need not be invertible.

Original: Using the language of topological conjugacy, we can describe a speculative but plausible interpretation of the results of Section 5.
Revised: Using the language of topological conjugacy, we can describe a speculative but plausible interpretation of the results of Section 5.

Original: In that section we observed that models from the same cluster had very similar dynamical features and performed similarly to the model average of the cluster.
Revised: In that section we observed that models from the same cluster had very similar dynamical features and performed similarly to the model average of the cluster.

Original: This suggests that for each model F n in the same cluster, one has
Revised: This suggests that for each model F n in the same cluster, we have

Original: [Equation]
Revised: [Equation]

Original: where θ is the centroid of the cluster to which F n belongs.
Revised: where θ is the centroid of the cluster to which F n belongs.

Original: Note that here we replaced θ n with θ , thus assuming both that L hidden is small and that θ n is sufﬁciently close to θ .
Revised: Note that here we replaced θ n with θ , thus assuming both that L hidden is small and that θ n is sufﬁciently close to θ .

Original: Second, making thespeculative hypothesis that there exists an inverse V − 1 n to the map V n ,we hypothesize that the map V n provides a topological conjugacy between the base model F n and the meta-model (cid:101) F θ evaluated at θ .
Revised: Second, making the hypothesis that there exists an inverse V − 1 n to the map V n , the map V n may provide a topological conjugacy between the base model F n and the meta-model (cid:101) F θ evaluated at θ .

Original: Assuming further that these hypotheses hold for all models in the cluster and using the transitivity of topological conjugacy, this suggests thatthe base models belonging to the same cluster are topologically conjugate to one another.
Revised: Assuming further that our assumptions hold for all models in the cluster, using the transitivity of topological conjugacy we would conclude that base models belonging to the same cluster are topologically conjugate to one another.

Original: This would justify the intuition suggested by Figure 8 that the meta-model ﬁnds the common topological structure between the base models in the cluster.
Revised: This would justify the intuition suggested by Figure 8 that D YNAMO clusters models according to commonalities of topological structures of dynamics.