G O WITH THE FLOW : A DAPTIVE C ONTROL FOR N EURAL ODE S
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite their elegant formulation and lightweight memory cost, neural ordinary differential equations (NODEs) suffer from known representational limitations.
In particular, the single ﬂow learned by NODEs cannot express all homeomorphisms from a given data space to itself, and their static weight parameterization restricts the type of functions they can learn compared to discrete architectures with layer-dependent weights.
Here, we describe a new module called neurallycontrolled ODE (N-CODE) designed to improve the expressivity of NODEs.
The parameters of N-CODE modules are dynamic variables governed by a trainable map from initial or current activation state, resulting in forms of open-loop and closed-loop control, respectively.
A single module is sufﬁcient for learning a distribution on non-autonomous ﬂows that adaptively drive neural representations.
We provide theoretical and empirical evidence that N-CODE circumvents limitations of previous NODEs models and show how increased model expressivity manifests in several supervised and unsupervised learning problems.
These favorable empirical results indicate the potential of using data- and activity-dependent plasticity in neural networks across numerous domains.
I NTRODUCTION
The interpretation of artiﬁcial neural networks as continuous-time dynamical systems has led to both theoretical and practical advances in representation learning.
According to this interpretation, the separate layers of a deep neural network are understood to be a discretization of a continuous-time operator so that, in effect, the net is inﬁnitely deep.
One important class of continuous-time models, neural ordinary differential equations (NODEs) (Chen et al., 2018), have found natural applications in generative variational inference (Grathwohl et al., 2019) and physical modeling (K ¨ ohler et al., 2019; Ruthotto et al., 2020) because of their ability to take advantage of black-box differential equation solvers and correspondence to dynamical systems in nature.
Nevertheless, NODEs suffer from known representational limitations, which researchers have tried to alleviate either by lifting the NODE activation space to higher dimensions or by allowing the transition operator to change in time, making the system non-autonomous (Dupont et al., 2019).
For example, Zhang et al. (2020) showed that NODEs can arbitrarily approximate maps from R d to R if NODE dynamics operate with an additional time dimension in R d +
and the system is afﬁxed with an additional linear layer.
The same authors showed that NODEs could approximate homeomorphisms from R d to itself if the dynamics were lifted to R 2 d .
Yet, the set of homeomorphisms from R d to itself is in fact quite a conservative function space from the perspective of representation learning, since these mappings preserve topological invariants of the data space, preventing them from “disentangling” data classes like those of the annulus data in Fig.
1 (lower left panel).
In general, much remains to be understood about the continuous-time framework and its expressive capabilities.
In this paper, we propose a new approach that we call neurally-controlled ODEs (N-CODE) designed to increase the expressivity of continuous-time neural nets by using tools from control theory.
Whereas previous continuous-time methods learn a single, time-varying vector ﬁeld for the whole input space, our system learns a family of vector ﬁelds parameterized by data.
We do so by mapping the input space to a collection of control weights which interact with neural activity to optimally steer model dynamics.
The implications of this new formulation are critical for model expressivity.
In particular, the transformation of the input space is no longer constrained to be a homeomorphism, since the ﬂows associated with each datum are speciﬁcally adapted to that point.
Consequently, our system can easily “tear” apart the two annulus classes in Fig.
1 (lower right panel) without directly lifting the data space to a higher dimension.
Moreover, when control weights are allowed to vary in time, they can play the role of fast, plastic synapses which can adapt to dynamic model states and inputs.
The rest of the paper proceeds as follows.
First, we will lay out the background for N-CODE and its technical formulation.
Then, we will demonstrate its efﬁcacy for supervised and unsupervised learning.
In the supervised case, we show how NCODE can classify data by learning to bifurcate its dynamics along class boundaries as well as memorize high-dimensional patterns in real-time using fast synapses.
Then, we show how the ﬂows learned by N-CODE can be used as latent representations in an unsupervised autoencoder, improving image generation over a base model.
B ACKGROUND
Neural ODEs (NODEs) (Chen et al., 2018) are dynamical systems of the form
[Equation]
where X is a space of features, θ ∈ Θ is a collection of learnable parameters, and f : X × Θ × R
(cid:55)→ X is an equation of motion which we take to be differentiable on its whole domain.
f deﬁnes a ﬂow , i.e. a triple ( X , R , Φ θ ) with Φ θ : X × R (cid:55)→ X deﬁned by
[Equation]
which relates an initial point x (0) to an orbit of points { x ( t )
=
Φ θ ( x (0) , t ) , t ∈ R } .
For a ﬁxed T , the map x (cid:55)→ Φ θ ( x, T ) is a homeomorphism from X to itelf parametrized by θ .
Several properties of such ﬂows make them appealing for machine learning.
For example, the ODEs that govern such such ﬂows can be solved with off-the shelf solvers and they can potentially model data irregularly sampled in time.
Moreover, such ﬂows are reversible maps by construction whose inverse is just the system integrated backward in time, Φ θ ( ., t ) − 1 =
Φ θ ( ., − t ) .
This property enables depth-constant memory cost of training thanks to the adjoint sensitivity method (Pontryagin
Lev Semyonovich ; Boltyanskii V G & F, 1962) and the modeling of continuous-time generative normalizing ﬂow algorithms ( ? ).
Interestingly, discretizing Eq.
2 yields the recursive formulation of a residual network (He et al., 2015) with a single residual operator f θ :
[Equation]
In this sense, NODEs with a time-independent (autonomous) equation of motion, f , are the inﬁnitely-deep limit of weight-tied residual networks.
Relying on the fact that every nonautonomous dynamical system with state x ∈ R d is equivalent to an autonomous system on the extended state ( x , t ) ∈ R d +1 , Eq.
2 can also be used to model general, weight-untied residual networks.
However it remains unclear how dependence of f in time should be modeled in practice and how their dynamics relate to their discrete counterparts with weights evolving freely across blocks through gradient descent.
N-CODE: L EARNING TO CONTROL DATA - DEPENDENT FLOWS
General formulation - The main idea of N-CODE is to consider the parameters, θ ( t ) , in Eq.as control variables for the dynamical state, x ( t ) .
Model dynamics are then governed by a coupled system of equations on the extended state z ( t ) =
( x ( t ) , θ ( t )) .
The initial value of the control weights, θ (0) , is given by a mapping γ : X → Θ .
Throughout, we assume that the initial time point is t = 0 .
The full trajectory of control weights, θ ( t ) , is then output by a controller, g , given by another differentiable equation of motion g : Θ
× X × R (cid:55)→ Θ with initial condition γ ( x 0 ) .
Given an initial point, x (0) , we can solve the initial value problem (IVP),
[Equation]
[Equation]
where h = ( f, g ) .
We may think of g and γ as a controller of the dynamical system with equation of motion, f .
We model g and γ as neural networks parameterized by µ ∈ R n µ and use gradient descent techniques where the gradient can be computed by solving an adjoint problem (Pontryagin Lev Semyonovich ; Boltyanskii V G & F, 1962) that we describe in the next section.
Our goal here is to use the metaparameterization in the space Θ to create richer dynamic behavior for a given f than directly optimizing ﬁxed weights θ .
Well-posedness - If f and g are continuously differentiable with respect to x and θ and continuous with respect to t , then, for all initial conditions ( x (0) , θ (0)) , there exists a unique solution z for Eq.
4 by the Cauchy-Lipschitz theorem.
This result leads to the existence and uniqueness of the augmented ﬂow ( X × Θ , R , Φ µ ) with Φ µ : ( X × Θ) × R
(cid:55)→ X × Θ .
Moreover, considering the restriction of such a ﬂow on X , we are now endowed with a universal approximator for at least the set of homeomorphisms on X given that this restriction constitutes a non-autonomous system.
We discuss now how g and γ affect the evolution of the variable x ( t ) , exhibiting two forms of control and noting how they relate to previous extensions of NODEs.
3. OPEN AND CLOSED - LOOP CONTROLLERS
If the controller outputs control weights as a function of the current state, x ( t ) , then we say it is a closed-loop controller.
Otherwise, it is an open-loop controller.
Open-loop control: First, we consider the effect of using only the mapping γ in Eq.
4 as a controller.
Here, γ maps the input space X to Θ so that f is conditioned on x (0) but not necessarily on x ( t ) for t > 0 .
In other words, each initial value x (0) evolves according to its own learned ﬂow ( X , R , Φ γ ( x (0) ) ) .
This allows for trajectories to evolve more freely than within a single ﬂow that must account for the whole data distribution and resolves the problem of non-intersecting orbits (see
Figure 3).
Recently, (Massaroli et al., 2020b) proposed a similar form of data-conditioned openloop control with extended state ( x ( t ) , x (0))) .
This is a version of our method in which γ is of the form γ ( x ) =
θ (0) =
[ C : id ] with C a constant vector, id is the identity function, and : denotes concatenation.
Our open-loop formulation makes an architectural distinction between controller and dynamics and is consequently generalizable to the following closed-loop formulation.
Closed-loop control:
Deﬁning a differentiable mapping, g , outputting the time-dependent control weights θ ( t ) given the state of the variable x ( t ) yields a non-autonomous system on X .
This can be seen as a speciﬁc dimension augmentation technique where additional variables correspond to the parameters θ .
However, contrary to appending extra dimensions which does not change the autonomous property of the system, this augmentation results in a module describing a time-varying transformation Φ θ ( t ) ( x 0 , t ) .
Note that this formulation generalizes the functional parameterization proposed in recent non-autonomous NODES systems (Choromanski et al., 2020; Massaroli et al., 2020b; ? ), since the evolution of θ ( t ) depends on x ( t ) .
Much like in the case of classical control theory, we hypothesized that the use of dynamic control weights would be of particular use in reacting to a non-stationary stimulus.
We evaluate this hypothesis in section.
The expressivity of N-CODE compared to other continuous-time neural networks is encapsulated in the following proposition.
Proposition 1 - There exists a transformation φ : R d → R d which can be expressed by N-CODE but not by NODEs.
In particular, φ is not a homeomorphism.
This result is proved in appendix, and shows that both open-loop and closed-loop control systems overcome NODEs’ expressivity constraint with two distinct strategies, data-conditioning and statespace augmentation.
3. T RAINING
Loss function: Dynamics are evaluated according to a generalized loss function that integrates a cost over some interval [0 , T ] :
[Equation]
The loss in Eq.
5 is more general than in Chen et al.
(2018) since (cid:96) can be any Lebesgue-measurable function of both states, x , and control parameters, θ .
In particular, this includes penalties at discrete time points or over the entire trajectory (Massaroli et al., 2020b) but also regularizations on the weights or activations over the entire trajectory rather than the ﬁnal state z ( T ) of the system.
In order to estimate a control function θ ( t ) that is optimal with respect to Eq. 5, we invoke Pontryagin’s maximum principle (PMP)
(Pontryagin Lev Semyonovich ; Boltyanskii V G & F, 1962), which only requires mild assumptions on the functional control space Θ and applies to functions f
that are non-smooth in θ .
The PMP gives necessary conditions on θ ( t ) at optimality via the augmented adjoint variable a ( t ) .
This quantity is the Jacobian of (cid:96) with respect to both x ( t ) and θ ( t ) .
In the case of θ ( t ) being differentiable with respect to the meta-parameters µ , solving for the augmented adjoint state a ( t ) as in Chen et al. (2018) allows us to compute the gradient of the loss with respect to µ thanks to Theorem 1.
Theorem 1 - Augmented adjoint method:
Given the IVP of equation 4 and for (cid:96) deﬁned in equation 5, we have:
[Equation]
such that a satisﬁes
[Equation]
where ∂h ∂ z is the Jacobian of h with respect to z :
[Equation]
In practice, we compute the Jacobian for this augmented dynamics with open source automatic differentiation libraries Pytorch (Paszke et al., 2019), enabling seamless integration of NCODE modules in architectures of such modules section B of
Appendix.
E XPERIMENTS 4. R EFLECTION AND C ONCENTRIC A NNULI
We introduce our approach with the 1- and 2-dimensional problems discussed in (Dupont et al., 2019; Massaroli et al., 2020b), consisting of learning either the reﬂection map ϕ ( x ) = − x or a linear classiﬁcation boundary on a data-set of concentric annuli.
Earlier results have shown that vanilla NODEs cannot learn these functions since NODEs preserve the topology of the data space, notably its linking number, leading to unstable and complex ﬂows for entangled classes.
However, both the open and closed loop formulations of N-CODE easily ﬁt these functions, as shown in Figs 3 and 4.
The adaptive parameterization allows N-CODE to learn separate vector ﬁelds for each input, allowing for simpler paths that ease model convergence.
Informally, we may think of the classiﬁcation decisions as being mediated by a bifurcation parameterized by the data space.
Transiting across the true classiﬁcation boundary switches the vector ﬁeld from one pushing points to one side of the classiﬁcation boundary to the other.
Moreover, the construction of γ and g as a neural network guarantees a smooth variation of the ﬂow with respect to the system state, which can potentially provide interesting regularization properties on the family learned by the model.
4. S UPERVISED IMAGE CLASSIFICATION
[Table]
We now explore different data-dependent controls for continuous-time neural network models on an image classiﬁcation task.
We tested the openloop version of N-CODE against augmented NODEs (Dupont et al., 2019) and the data-control model of (Massaroli et al., 2020b).
We perform this experiment on the MNIST and CIFAR-10 image datasets.
Our model consists of the dynamical system with equation of motion f expressed as a single convolutional block (1 × 1 (cid:55)→ 3 × 3 (cid:55)→ 1 × 1) with 70 channels.
The control weights θ ∈ R K are a real-valued vector of K concatenated parameters instantiating the convolution kernels.
The 10-class prediction vector is obtained as a linear transformation of the ﬁnal state x ( T ) of the dynamics with a softmax activation.
We aimed at simplicity by deﬁning the mapping γ as a two-layer perceptron with 10 hidden units.
We varied how γ affects θ by deﬁning different linear heads for each convolution in the block.
(see C.2 of Appendix).
We did not perform any ﬁne-tuning, although we did augment the state dimension, akin to ANODE, by increasing the channel dimension by 20, which helped stabilize training.
We show that the open-loop control outperforms competing models but also trains more quickly.
Results are shown in Table.with further details in Section C.2 of Appendix.
4. R EAL - TIME PATTERN MEMORIZATION
To validate our intuition that closed-loop control is particularly valuable on problems involving ﬂexible adaptation to non-stationary stimuli and to distinguish our system from related data-conditioned methods, we trained N-CODE on a dynamic, few-shot memorization task (Miconi et al., 2018).
Here, the model is cued to quickly memorize sets of sequentially presented n -dimensional binary patterns and to reconstruct one of these patterns when exposed to a degraded version.
For each presentation episode, we exposed the model sequentially to m = 2 , 5 , or 10 randomly generated n -length ( n = 100 , 1000 ) bit patterns with {− 1 , 1 } -valued bits.
Then, we presented a degraded version of one of the previous patterns in which half of the bits were zeroed, and we tasked the system with reconstructing this degraded pattern via the dynamic state, x ( t ) .
This is a difﬁcult task since it requires the memorization of several high-dimensional data in real-time.
Miconi et al.
(2018) have shown that, although recurrent networks are theoretically able to solve this task, they struggle during learning, while models storing transient information in dynamic synapses tend to perform better.
Following this result, we evaluated the performance of a closed-loop N-CODE model in which the output of the control function takes the form of dynamic synapses (Eq. 7).
Here, the control parameters are a matrix of dynamic weights, θ
( t ) ∈
R m × m , which are governed by the outer product µ (cid:12) xx T ,
[Equation]
where µ is matrix of learned parameters, θ 0 is a learned initial condition, ρ is an element-wise sigmoidal non-linearity, (cid:12) is element-wise multiplication, and x stim is the presented stimulus.
We have suppressed t for concision.
We ran this experiment on the closed-loop version of N-CODE as well as an LSTM, continuous-time ODE-RNN (Rubanova et al., 2019) and the data-controlled model of Massaroli et al.
(2020b).
All systems had the same number of learnable parameters, except for the LSTM which was much larger.
We found that all considered models, except closed-loop NCODE, struggled to learn this task (Fig. 5).
Note that chance is .25 since half the bits are degraded.
A discrete LSTM endowed with a much larger hidden state of 5000 neurons learns marginally better than chance level.
ODE-RNN and data control versions plateau at 17.5% error rate.
For both n tested, N-CODE learned the problem not only much better, but strikingly faster.
4. U NSUPERVISED LEARNING : I MAGE AUTOENCODING WITH CONTROLLED FLOW
Finally, we use our control approach in an autoencoding model for low-dimensional representation learning in order to see if merely adding N-CODE to a simpler model can improve performance.
Our idea is to use the solution to a system of linear ODEs to encode the model’s latent representations.
First, images from U ∈ R n are mapped by an encoder to a space of couplings, θ , by
[Equation]
The latent code is taken as the vector x ( T ) acquired by solving the autonomous differential system presented in Eq.
9 from 0 to T starting from the initial condition ( x 0 , γ µ ( u )) with x 0 ∼ N (0 , I ) :
[Equation]
This model, which we call autoN-CODE, can be interpreted as a hybrid version of autoencoders and generative normalizing ﬂows, where the latent generative ﬂow is data-dependent and parameterized by the encoder output.
We found that a simple linear system was already more expressive than a discrete linear layer in shaping the latent representation.
Since the output dimension of the encoder grows quadratically with the desired latent code dimension of the bottleneck, we adopted a sparse prediction strategy, where as few as two elements of each row of θ were non-zero, making our model match exactly the number of parameters of a VAE with same architecture.
We believe that the increase in representational power over a vanilla autoencoder, as shown by improvements in Fr ´ echet Inception Distance (FID; Heusel et al. and more disentangled latent representations (see A.7), despite these simplifying assumptions, shows the potential of this approach for representation learning (Fig. 2).
Training details can be found in Sec.
C.4.
4. I MAGE GENERATION
FID (
[Equation]
CIFARCelebA
[Table]
[Table]
Here we measure the marginal improvement acquired by adding latent ﬂows to a baseline VAE.
In order to endow our deterministic model with the ability to generate image samples, we perform, similarly to (Ghosh et al., 2020), a post hoc density estimation of the distribution of the latent trajectories’ ﬁnal state.
Namely, we employ a gaussian mixture model ﬁt by expectation-maximization and explore the effect of increasing the number of components in the mixture.
We report in Table 2, contrary to Ghosh et al.
(2020), a substantial decrease in FID of our sampled images to the image test set with an increasing number of components in the mixture (see Fig.
6 for latent evolution and Fig. 7 for sampled examples).
These results suggest, considering the identical architecture of the encoder and decoder to a vanilla model, that the optimal encoding control formulation produced a structural change in the latent manifold organization.
(See section C.7 in Appendix).
Finally, we note that this model offers several original possibilities to explore, such as sampling from the control coefﬁcients, or adding a regularization on the dynamic evolution as in (Finlay et al., 2020), which we leave for future work.
R ELATED WORK
Neural ODEs - This work builds on the recent development of Neural ODEs, which have shown great promise for building generative normalizing ﬂows (Grathwohl et al., 2019) used for modeling molecular interactions (K¨ohler et al., 2019) or in mean ﬁeld games (Ruthotto et al., 2019).
More recent work has focused on the topological properties of these models (Dupont et al., 2019; Zhang et al., 2020), introducing time-dependent parameterization (Zhang et al., 2019b; Massaroli et al., 2020b; Choromanski et al., 2020), developing novel regularization methods using optimal transport or stochastic perturbations (Finlay et al., 2020; Oganesyan et al., 2020) and adapting them to stochastic differential equations (Tzen & Raginsky, 2019).
Optimal Control - Several authors have interpreted deep learning optimization as an optimal control problem (Li et al., 2017; Benning et al., 2019; Liu & Theodorou, 2019; Zhang et al., 2019a;
Seidman et al., 2020) providing strong error control and yielding convergence results for alternative algorithms to standard gradient-based learning methods.
Of particular interest is the mean-ﬁeld optimal control formulation of (E et al., 2018), which notes the dependence of a unique control variable governing the network dynamics on the whole data population.
Hypernetworks - Finally, our system is related to network architectures with adaptive weights such as hypernetworks (Ha et al., 2017), dynamic ﬁlters (Jia et al., 2016) and spatial transformers (Jaderberg et al., 2015).
Though not explicitly formulated as neural optimal control, these approaches effectively implement a form of modulation of neural network activity as a function of input data and activation state, resulting in compact and expressive models.
These methods demonstrate, in our opinion, the signiﬁcant untapped potential value in developing dynamically controlled modules for deep learning.
C ONCLUSION
In this work, we have presented an original control formulation for continuous-time neural feature transformations.
We have shown that it is possible to dynamically shape the trajectories of the transformation module applied to the data by augmenting the network with a trained control mechanism, and further demonstrated that this can be applied in the context of supervised and unsupervised representation learning.
In future work, we would like to investigate the robustness and generalization properties of such controlled models as well as their similarities with fast-synaptic modulation systems observed in neuroscience, and test this on natural applications such as recurrent neural networks and robotics.
An additional avenue for further research is the connection between our system and the theory of bifurcations in dynamical systems and neuroscience.
A PPENDIX A T HEORETICAL RESULTS A. P ROOF OF P ROPOSITION
In order to show that the set of functions deﬁned by N-CODE are not necessarily homeomorphisms on X , we show that they are not generally injective.
Open-loop : Consider the one-dimensional N-CODE system Φ : ( X , R )
(cid:55)→ X with equation of motion f ( x , θ ) =
− θ where θ = γ ( x (0))
=
x (0) .
This system has solution
[Equation]
However, this implies
[Equation]
giving the result.
Closed-loop : Similarly, we can show that the following 2D oscillatory system:
[Equation]
But Φ( x, π/ 2) = 0 , so x (cid:55)→ Φ( x , t ) =
x ( t ) is not injective.
This shows that the two control forms, open and closed, are not restricted to learn homemorphisms on the space X .
A. P ROOF OF T HEOREM
Proof.
The proof is inspired from Massaroli et al.
(2020a).
We place our analysis in the Banach space R n .
Since (cid:96) and h are continuously differentiable, let us form the Lagrangian functional L where a is an element of the dual space of z = (cid:20)
[Equation]
[Equation]
The constraint (cid:104) a ( t ) , ∂z ∂t − h ( z , t ) (cid:105) is always active on the admissible set by construction of z in equation 4, such that we have ∀ z , a , ∂ L ∂ µ ( z , a , t )
=
∂l
∂ µ ( z , a , t ) .
Moreover, integrating the left part of the integral in equation 13 gives:
[Equation]
[Equation]
Now, given that (cid:96) − (cid:104) d a dt , z (cid:105) is differentiable in µ for any t ∈ [0 , T ] , Leibniz integral rule allows to write
[Equation] [Equation]
The last equation can be reordered as:
[Equation]
Posing that a ( T ) =
O |X| + | Θ | and (cid:104)
d a dt − a ∂h ∂ z − ∂(cid:96)
∂ z , ∂ z ∂ µ (cid:105) = 0 , the result follows.
Remark 1: This result allows to compute gradient estimate assuming that µ directly parametrizes h such that the gradient ∂h ∂ µ are straightforward to compute.
However, in the case of optimizing the initial mapping γ that infer the variable θ , this result can be combined with usual chain rule to estimate the gradient:
[Equation] [Equation]
B I MPLEMENTATION
Generic N-CODE module - We provide here a commented generic PyTorch implementation for the N-CODE module in the open-loop setting.
class NCODE_func (torch.nn.
Module): def __init__(self, *args): super(NCODE_func
[Equation]
self
[Equation]
__init
[Equation]
() #Definition of the ODE system self.f
= ... self.g
= ...
def forward(self, t, z): #Unpack variables (theta_t is potentially a tuple of tensors)
x_t, *theta_t =
z #Compute evolution od the system d_x_t = self.f(x_t,theta_t) d_theta_t
= self.g(theta_t,d_x_t = f(x_t,theta_t) return (d_x_t, *d_theta_t) class NCODE_block (torch.nn.
Module): def __init__(self, *args): super(NCODE_block, self).__init__() #Definition of module and controller self.gamma = ... self.NCODE_func = ...
def forward(self, x_0):
#Set initial conditions self.theta_0 = self.gamma(x_0) z_0 = (x_0,*self.theta_0)
#Run the system forward with defined settings x_t, *theta_t =
odeint_adjoint(self.odefunc, z_0, integration_time, rtol, atol, method,options, adjoint_params=self.
NCODE_func.parameters()) return (x_t, *theta_t)
C E XPERIMENTAL RESULTS
Resources - Our experiments were run on a 12GB NVIDIA® Titan Xp GPUs cluster equipped with CUDA 10.1 driver.
Neural ODEs were trained using the torchdiffeq (Chen et al., 2018)
PyTorch package.
[Figure]
C. S UPERVISED IMAGE CLASSIFICATION
For MNIST and CIFAR-10 classiﬁcation, we deﬁned the equation of motion f as sequence of convolutional ﬁlters replicating a ResNet block architecture with padding to conserve dimensionality and non-linear ReLU activation.
The base architecture takes one of the forms
• 1 × 1 , k ﬁlters, 1 stride, 0 padding • 3 × 3 , k ﬁlters, 1 stride, 1 padding • 1 × 1 , c + a ﬁlters, 1 stride, 0 padding
with k the number of ﬁlters, c the number of channels (1 for MNIST and 3 for CIFAR10) and a the augmentation channel (Dupont et al., 2019).
The open-loop controller is a very simple linear transformation of the data into 10 hidden units followed by multiple heads outputting the vector of weights for each convolution kernel.
In order to examine the effect of the size of dimensionality of the equation of motion on learnability, we tried different parameterizations for f :
• Full : The three kernels are conditioned by the controller γ .
• 3 × 3 : The inner 3x3 kernels are dynamically conditioned.
• 1 × 1 : The 1 × 1 kernels are dynamically conditioned.
• Data control : This corresponds to the form of data-control proposed by (Massaroli et al., 2020b) where the image data is concatenated with state x ( t ) at each evaluation of the ﬁrst 1x1 convolution kernel.
All parameters not adaptively controlled are simply learned as ﬁxed parameters.
For all models we used the adaptative solver Dormund-Prince with tolerance of 1e-3.
The results show that N-CODE models achieve higher accuracy in fewer epochs and overall lower loss (Fig.
A.2, top panels). However, the simplicity of the mapping γ in this experiment and the absence of regularization of the induced map Φ γ (Finlay et al., 2020) yields a complex ﬂow requiring a higher total number of function evaluations (Fig. A.2, lower panels).
This potentially prevents NCODE from achieving even better accuracy.
Interestingly, the results for the partially controlled systems suggest that gradient descent accommodates the learning of hybrid modules f with both static and dynamic convolution parameters.
These results advocate for the exploration of speciﬁc closed-loop formulations and regularization of control for convolution layers which we leave for future work.
C. P ATTERN M EMORIZATION
This experiment is an update of Miconi et al. (2018), which showed that using a simple Hebbian update rule between weights endowed recurrent networks with fast memory properties.
The base model is a single-cell recurrent network with all-to-all coupling, θ ( t ) .
The considered models are
Open-loop control : We adopt the implementation of (Massaroli et al., 2020b) by appending the presented pattern x stim to the dynamic variable x ( t ) such that θ ( t )
= θ is linear layer L ( R 2 × N , R N ) .
Closed-loop control : The weights are fully dynamic and the inﬂuence of the plastic evolution is tuned by learning the components µ that apply an element-wise gain for every weight velocity, according to Eq.
ODE-RNN :
Here, a static-weight NODE models the evolution of a continuous-time single-cell recurrent network whose hidden state is carried over from one presentation to the next.
A supplementary linear read-out L ( R N , R N ) outputs the model’s guess and balances the number of
parameters with other models. LSTM :
A vanilla LSTM cell (Hochreiter & Schmidhuber, 1997).
In coherence with (Miconi et al., 2018) results, the hidden state dimension need to be greatly increased for the module to start memorizing.
We tested a 5000-dimensional hidden state for the reported results.
For each model, we learn the model weights with gradient descent using an Adam optimizer with a learning rate λ
= 3 e − 4 and use the L 2 distance between the reconstruction and objective pattern as our objective function.
For continuous models, each episode consists of a sequential presentation of each pattern for 0.5 sec followed by a query time of 0.5 sec.
For the LSTM, we adopted the setting of (Miconi et al., 2018) where the sequence of presentation is discretized into 5 time-steps of presentations.
We tested for episodes of 3,5 and 10 patterns with 2 presentations in random order and different proportions of degradation (0.5,0.7 and 0.9).
Static-weight model performance rapidly deteriorated in more challenging settings, whereas N-CODE reconstruction converged to a residual error below 1% in all cases.
C. A UTO N-CODE ARCHITECTURES
MNIST
CIFARCelebA
[Equation]
Encoder :
Flatten
Conv 256 (cid:55)→ BN (cid:55)→ Relu Conv 256 (cid:55)→ BN (cid:55)→ Relu
FC 400 (cid:55)→ Relu
Conv 512 (cid:55)→ BN (cid:55)→ Relu Conv 512 (cid:55)→ BN (cid:55)→ Relu
FC 25 (cid:55)→ Norm
Conv 1024 (cid:55)→ BN (cid:55)→ Relu Conv 1024 (cid:55)→ BN (cid:55)→ Relu
FC 128 ∗ 2 (cid:55)→ Norm
FC 64 ∗ 10 (cid:55)→ Norm
Latent dynamics:
ODE[0,10]
ODE[0,10] ODE[0,10]
Decoder :
FC 400 (cid:55)→ Relu
FC 1024 ∗ 8 ∗ 8 (cid:55)→ Norm
FC 1024 ∗ 8 ∗ 8 (cid:55)→ Norm FC 784 (cid:55)→
Sigmoid
ConvT 512 (cid:55)→ BN (cid:55)→ Relu ConvT 512 (cid:55)→ BN (cid:55)→
[Table]
The dynamic is run on the time segment [0,10] which empirically yields good results.
Adapting the models used by Tolstikhin et al.
(2017) and Ghosh et al.
(2020), we use a latent space dimension of 25 for MNIST, 128 for CIFAR-10 and 64 for CelebA.
All convolutions and transposed convolutions have a ﬁlter size of 4 × 4 for MNIST and CIFAR-10 and 5 × 5 for CELEBA.
We apply batch normalization to all layers.
They all have a stride of size 2 except for the last convolutional layer in the decoder.
We use Relu non-linear activation and batch normalisation at the of convolution train and test splits are used for the three datasets.
For training, use a mini-batch size of 64 in MNIST and CIFAR and 16 for CelebA in AutoN-CODE.
(64 for control models.)
All models are trained for a maximum of 50 epochs on MNIST and CIFAR and 40 epochs on CelebA.
We make no additional change to the decoder.
We train the parameters of the encoder and decoder module for minimizing the mean-squared error (MSE) on CIFAR-10 and CelebA (Liu et al., 2015) or alternatively the Kullback-Leibler divergence between the data distribution and the output of the decoder for MNIST (formulas in Appendix.)
Gradient descent is performed forepochs with the Adam optimizer (Kingma & Ba, 2014) with learning rate λ = 1 e − 3 reduced by half every time the loss plateaus
C. V ISUALIZATION OF LATENT CODE DYNAMICAL EVOLUTION C. E XPLORING SAMPLING DISTRIBUTIONS
For random sampling, we train the VAE with a N (0 , I ) prior.
For the deterministic models, samples are drawn from a mixture of multivariate gaussian distributions ﬁt
using the testing set embeddings.
The distribution is obtained through expectation-maximization Dempster et al.
(1977) with one single k-means initialization, tolerance 1 e −
and run for at most 100 iterations.
We compute the FID using 10k generated samples evaluated against the whole test set for all FID evaluations, using the standard 2048-dimensional ﬁnal feature vector of an Inception V3 following Heusel et al.
(2017) implementation.
The results show that 10K components (matching the number of data points in the test sets) naturally overﬁts on the data distribution.
Generated images display marginal changes compared to test images.
However, 1000 components does not, showing that our AutoN-CODE sampling strategy mediates a trade-off between sample quality and generalization of images.
We alternatively tested non-parametric kernel density estimation with varying kernel variance to replace our initial sampling strategy.
We report similar results to the gaussian mixture experiment with an overall lower FID of
AutoN-CODE for small variance kernels.
As the ﬁtting distribution becomes very rough ( σ ≈ 5 ), the generated image quality is highly deteriorated.(see Figure A6)
C. A UTO N-CODE L ATENT C LUSTERING
We further investigate the representations learned by our model by measuring its clustering accuracy against several other techniques in Fig.
A.10.
Similarly to the image generation experiment, we consider the ﬁnal state of the dynamical system as the latent representation that we cluster using a 10 component gaussian mixture with different initializations.
We did not perform any supplementary ﬁne-tuning or data augmentation to train our model, but we also tested a version with further dimensionality reduction using t-SNE (Maaten & Hinton, 2008).
The results, although inferior to recent deep clustering techniques, show better clustering accuracy than other autoencoding models, suggesting a different organization of the latent code compared to the vanilla linear projection of the autoencoder.
C. L ATENT CODE INTERPOLATION
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]