Original: D EEP C
Revised: D EEP C

Original: ONTINUOUS
Revised: ONTINUOUS

Original: N ETWORKS
Revised: N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: CNNs and computational models of biological vision share some fundamental principles, which, combined with recent developments in deep learning, have opened up new avenues of research in neuroscience.
Revised: CNNs and computational models of biological vision share some fundamental principles, which, combined with recent developments in deep learning, have opened up new avenues of research in neuroscience.

Original: However, in contrast to biological models, conventional CNN architectures are based on spatio-temporally discrete representations, and thus cannot accommodate certain aspects of biological complexity such as continuously varying receptive ﬁeld sizes and temporal dynamics of neuronal responses.
Revised: However, in contrast to biological models, conventional CNN architectures are based on spatio-temporally discrete representations, and thus cannot accommodate certain aspects of biological complexity such as continuously varying receptive ﬁeld sizes and temporal dynamics of neuronal responses.

Original: Here we propose deep continuous networks (DCNs), which combine spatially continuous convolutional ﬁlter representations, with the continuous time framework of neural ODEs.
Revised: Here we propose deep continuous networks (DCNs), which combine spatially continuous convolutional ﬁlter representations, with the continuous time framework of neural ODEs.

Original: This allows us to learn the spatial support of the ﬁlters during training, as well as model the temporal evolution of feature maps, linking DCNs closely to biological models.
Revised: This allows us to learn the spatial support of the ﬁlters during training, as well as model the temporal evolution of feature maps, linking DCNs closely to biological models.

Original: We show that DCNs are versatile.
Revised: We show that DCNs are versatile.

Original: Experimentally, we demonstrate their applicability to a standard classiﬁcation problem, where they allow for parameter reductions and meta-parametrization.
Revised: Experimentally, we demonstrate their applicability to a standard classiﬁcation problem, where they allow for parameter reductions and meta-parametrization.

Original: We illustrate the biological plausibility of the scale distributions learned by DCNs and explore their performance in a pattern completion task, which is inspired by models from computational neuroscience.
Revised: We illustrate the biological plausibility of the scale distributions learned by DCNs and explore their performance in a pattern completion task, which is inspired by models from computational neuroscience.

Original: Finally, we suggest that the continuous representations learned by DCNs may enable computationally efﬁcient implementations.
Revised: Finally, we suggest that the continuous representations learned by DCNs may enable computationally efﬁcient implementations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Computational neuroscience and computer vision have a long and mutually beneﬁcial history of cross-pollination of ideas (Sejnowski, 2020; Cox & Dean, 2014).
Revised: Computational neuroscience and computer vision have a long and mutually beneﬁcial history of cross-pollination of ideas (Sejnowski, 2020; Cox & Dean, 2014).

Original: The current state-of-the-art in computer vision relies heavily on deep neural networks (DNNs), and in particular convolutional neural networks (CNNs), from which multiple analogies can be drawn to biological circuits (Kietzmann et al., 2018).
Revised: The current state-of-the-art in computer vision relies heavily on deep neural networks (DNNs), and in particular convolutional neural networks (CNNs), from which multiple analogies can be drawn to biological circuits (Kietzmann et al., 2018).

Original: Speciﬁcally, recent advances in DNNs have enabled researchers to learn more accurate models of the response properties of neurons in the visual cortex (Klindt et al., 2017; Cadena et al., 2019; Ecker et al., 2019), as well as to test decades old hypotheses from neuroscience in the domain of computer vision (Lindsey et al., 2019).
Revised: Speciﬁcally, recent advances in DNNs have enabled researchers to learn more accurate models of the response properties of neurons in the visual cortex (Klindt et al., 2017; Cadena et al., 2019; Ecker et al., 2019), as well as to test decades old hypotheses from neuroscience in the domain of computer vision (Lindsey et al., 2019).

Original: However, contrary to biological models,
Revised: However, contrary to biological models,

Original: CNNs typically operate in the domain of spatio-temporally discrete signals, and employ appropriately discretized kernels, as a natural part of digital image processing.
Revised: CNNs typically operate in the domain of spatio-temporally discrete signals, and employ appropriately discretized kernels, as a natural part of digital image processing.

Original: In computational neuroscience, on the other hand, large scale neural network models of the visual system often adopt continuous, closed-form expressions to describe spatio-temporal receptive ﬁelds, as well as the interaction strength between populations of neurons (Dayan & Abbott, 2001).
Revised: In computational neuroscience, on the other hand, large scale neural network models of the visual system often adopt continuous, closed-form expressions to describe spatio-temporal receptive ﬁelds, as well as the interaction strength between populations of neurons (Dayan & Abbott, 2001).

Original: Among others, such descriptions serve to limit the scope and parameter space of a model, by utilizing prior information regarding receptive ﬁeld shapes (Jones & Palmer, 1987) and principles of perceptual grouping (Li, 1998).
Revised: Among others, such descriptions serve to limit the scope and parameter space of a model, by utilizing prior information regarding receptive ﬁeld shapes (Jones & Palmer, 1987) and principles of perceptual grouping (Li, 1998).

Original: In addition, the choice of continuous—and often analytic—functions help retain some analytical tractability in complex models involving a large number of coupled populations.
Revised: In addition, the choice of continuous—and often analytic—functions help retain some analytical tractability in complex models involving a large number of coupled populations.

Original: Our approach draws inspiration from such computational models to propose continuous representations of receptive ﬁelds in CNNs, where both the shape and the scale of the ﬁlters are trainable in the continuous domain.
Revised: Our approach draws inspiration from such computational models to propose continuous representations of receptive ﬁelds in CNNs, where both the shape and the scale of the ﬁlters are trainable in the continuous domain.

Original: In a complementary fashion, recent inﬂuential work in deep learning has introduced neural ordinary differential equations (ODEs) (Lu et al., 2018; Ruthotto & Haber, 2019; Chen et al., 2018) which propose a continuous time (or depth) interpretation of CNNs.
Revised: In a complementary fashion, recent inﬂuential work in deep learning has introduced neural ordinary differential equations (ODEs) (Lu et al., 2018; Ruthotto & Haber, 2019; Chen et al., 2018) which propose a continuous time (or depth) interpretation of CNNs.

Original: Such continuous time models both offer end-to-end training capabilities with backpropagation which are highly applicable to computer vision problems (e.g. by way of adopting ResNet blocks (He et al., 2016)), as well as help bridge the gap to computational biology where networks are often modelled as dynamical systems which evolve according to differential equations.
Revised: Such continuous time models both offer end-to-end training capabilities with backpropagation which are highly applicable to computer vision problems (e.g. by way of adopting ResNet blocks (He et al., 2016)), as well as help bridge the gap to computational biology where networks are often modelled as dynamical systems which evolve according to differential equations.

Original: In this work we aim to extend the impetus of the continuous time neural ODEs to the spatio-temporal domain.
Revised: In this work we aim to extend the impetus of the continuous time neural ODEs to the spatio-temporal domain.

Original: To that end we introduce deep continuous networks (DCNs), which are spatio-temporally continuous in that the neurons have spatially well-deﬁned receptive ﬁelds based on scale-spaces and Gaussian derivatives (Florack et al., 1996) and their activations evolve according to equations of motion comprising convolutional layers.
Revised: To that end we introduce deep continuous networks (DCNs), which are spatio-temporally continuous in that the neurons have spatially well-deﬁned receptive ﬁelds based on scale-spaces and Gaussian derivatives (Florack et al., 1996) and their activations evolve according to equations of motion comprising convolutional layers.

Original: We combine spatial and temporal continuity in a network with neural
Revised: We combine spatial and temporal continuity in a network with neural

Original: ODEs by learning linear weights for a set of analytic basis functions (as opposed to pixel-based weights), which can also intuitively be parametrized as a function of time, or network depth.
Revised: ODEs by learning linear weights for a set of analytic basis functions (as opposed to pixel-based weights), which can also intuitively be parametrized as a function of time, or network depth.

Original: The following outlines our main contributions: (i)
Revised: The following outlines our main contributions: (i)

Original: We provide a theoretical formulation of spatiotemporally continuous deep networks building on Gaussian derivative basis functions and neural
Revised: We provide a theoretical formulation of spatiotemporally continuous deep networks building on Gaussian derivative basis functions and neural

Original: ODEs; (ii) We demonstrate the applicability of DCN models, namely, that they exhibit a reduction in parameters, and can be used to parametrize convolutional ﬁlters as a function of time in a straightforward fashion, while achieving performance comparable with or better than ResNet and ODE-Net baselines; (iii)
Revised: ODEs; (ii) We demonstrate the applicability of DCN models, namely, that they exhibit a reduction in parameters, and can be used to parametrize convolutional ﬁlters as a function of time in a straightforward fashion, while achieving performance comparable with or better than ResNet and ODE-Net baselines; (iii)

Original: We show that ﬁlter scales learned by DCNs are consistent with biological observations and we propose that the combination of our design choices for spatial and temporal continuity may be helpful in studying the emergence of biological receptive ﬁeld properties as well as highlevel phenomena such as pattern completion; (iv) We suggest that the continuous representations learned by DCNs may be leveraged for computational savings.
Revised: We show that ﬁlter scales learned by DCNs are consistent with biological observations and we propose that the combination of our design choices for spatial and temporal continuity may be helpful in studying the emergence of biological receptive ﬁeld properties as well as highlevel phenomena such as pattern completion; (iv) We suggest that the continuous representations learned by DCNs may be leveraged for computational savings.

Original: We believe DCNs can bring together two communities as they provide a test bed for hypotheses and predictions pertaining to both biological systems as well as pushing the boundaries of biologically inspired computer vision.
Revised: We believe DCNs can bring together two communities as they provide a test bed for hypotheses and predictions pertaining to both biological systems as well as pushing the boundaries of biologically inspired computer vision.

Original:  D EEP C ONTINUOUS N ETWORKS 2. N EUROSCIENTIFIC MOTIVATION
Revised:  D EEP C ONTINUOUS N ETWORKS 2. N EUROSCIENTIFIC MOTIVATION

Original: There is little doubt that modern deep learning frameworks will be conducive to effective and insightful collaborations between neuroscience and machine learning (Richards et al., 2019).
Revised: There is little doubt that modern deep learning frameworks will be conducive to effective and insightful collaborations between neuroscience and machine learning (Richards et al., 2019).

Original: In particular in vision research, CNNs are becoming increasingly popular for modelling early visual areas (Batty et al., 2017; Ecker et al., 2019; Lindsey et al., 2019).
Revised: In particular in vision research, CNNs are becoming increasingly popular for modelling early visual areas (Batty et al., 2017; Ecker et al., 2019; Lindsey et al., 2019).

Original: Here we propose a model which can facilitate such investigations by linking the end-to-end trainable but discrete CNN architectures with the biologically more plausible and spatio-temporally continuous computational models.
Revised: Here we propose a model which can facilitate such investigations by linking the end-to-end trainable but discrete CNN architectures with the biologically more plausible and spatio-temporally continuous computational models.

Original: Structured receptive ﬁelds.
Revised: Structured receptive ﬁelds.

Original: Classical receptive ﬁelds (RFs) of cortical neurons display complex response properties with a wide array of selectivity structures already at early visual areas (Van den
Revised: Classical receptive ﬁelds (RFs) of cortical neurons display complex response properties with a wide array of selectivity structures already at early visual areas (Van den

Original: Bergh et al., 2010).
Revised: Bergh et al., 2010).

Original: Such response properties may also vary greatly based on multiple factors.
Revised: Such response properties may also vary greatly based on multiple factors.

Original: For example the RF size (spatial extent) is known to depend on eccentricity (Harvey & Dumoulin, 2011) and visual area (Smith et al., 2001) and may even change with depth within a cortical layer (Bauer et al., 1999).
Revised: For example the RF size (spatial extent) is known to depend on eccentricity (Harvey & Dumoulin, 2011) and visual area (Smith et al., 2001) and may even change with depth within a cortical layer (Bauer et al., 1999).

Original: Similarly, studies have shown that receptive ﬁeld size and spatial frequency selectivity of neurons may co-vary with input contrast (Sceniak et al., 2002).
Revised: Similarly, studies have shown that receptive ﬁeld size and spatial frequency selectivity of neurons may co-vary with input contrast (Sceniak et al., 2002).

Original: Based on these observations, we aim to build a model which can accommodate the biological complexity better than conventional CNNs, by explicitly modelling the RF size as a trainable parameter.
Revised: Based on these observations, we aim to build a model which can accommodate the biological complexity better than conventional CNNs, by explicitly modelling the RF size as a trainable parameter.

Original: To that end, we adopt a Gaussian scale-space representation for the convolutional ﬁlters, which we call structured receptive ﬁelds (SRFs) (Jacobsen et al., 2016).
Revised: To that end, we adopt a Gaussian scale-space representation for the convolutional ﬁlters, which we call structured receptive ﬁelds (SRFs) (Jacobsen et al., 2016).

Original: Previously, Gaussian scale-spaces have been proposed as a plausible model of biological receptive ﬁelds and feature extraction in low-level vision (Florack et al., 1992; Lindeberg & Florack, 1994; Lindeberg, 1993).
Revised: Previously, Gaussian scale-spaces have been proposed as a plausible model of biological receptive ﬁelds and feature extraction in low-level vision (Florack et al., 1992; Lindeberg & Florack, 1994; Lindeberg, 1993).

Original: Here, we are inspired partially by computational models which investigate the origin of response properties in the visual system, by employing RFs and recurrent interaction functions which scale as a difference of
Revised: Here, we are inspired partially by computational models which investigate the origin of response properties in the visual system, by employing RFs and recurrent interaction functions which scale as a difference of

Original: Gaussians (Somers et al., 1995; Ernst et al., 2001).
Revised: Gaussians (Somers et al., 1995; Ernst et al., 2001).

Original: Partially, we are motivated by the success of algorithms which utilize Gaussian scale-spaces (Lowe, 2004).
Revised: Partially, we are motivated by the success of algorithms which utilize Gaussian scale-spaces (Lowe, 2004).

Original: Neural ODEs.
Revised: Neural ODEs.

Original: Studies have shown that both the contrast (Albrecht et al., 2002) and spatial frequency (Frazor et al., 2004) response functions of cortical neurons display characteristic temporal proﬁles.
Revised: Studies have shown that both the contrast (Albrecht et al., 2002) and spatial frequency (Frazor et al., 2004) response functions of cortical neurons display characteristic temporal proﬁles.

Original: However, temporal dynamics are not incorporated into typical feed-forward CNN models.
Revised: However, temporal dynamics are not incorporated into typical feed-forward CNN models.

Original: In addition, it has been suggested that lateral interactions play an important role in the generation of complex and selective neuronal responses (Angelucci & Bressloff, 2006).
Revised: In addition, it has been suggested that lateral interactions play an important role in the generation of complex and selective neuronal responses (Angelucci & Bressloff, 2006).

Original: Such activity dynamics are often computationally modeled using recurrently coupled neuronal populations whose activations evolve according to coupled differential equations (Ben-Yishai et al., 1995; Ernst et al., 2001).
Revised: Such activity dynamics are often computationally modeled using recurrently coupled neuronal populations whose activations evolve according to coupled differential equations (Ben-Yishai et al., 1995; Ernst et al., 2001).

Original: [Figure]
Revised: [Figure]

Original: For temporally continuous computational representations consistent with biological models, we adopt the framework of neural ODEs (Chen et al., 2018).
Revised: For temporally continuous computational representations consistent with biological models, we adopt the framework of neural ODEs (Chen et al., 2018).

Original: Neural ODE interpretation of ResNet models presents an opportunity to explicitly model the temporal dynamics of feature extractors in DNNs.
Revised: Neural ODE interpretation of ResNet models presents an opportunity to explicitly model the temporal dynamics of feature extractors in DNNs.

Original: Under certain assumptions, neural ODEs can be interpreted as recurrent interactions (Rousseau et al., 2019), which are biologically plausible (Liao & Poggio, 2016).
Revised: Under certain assumptions, neural ODEs can be interpreted as recurrent interactions (Rousseau et al., 2019), which are biologically plausible (Liao & Poggio, 2016).

Original: Unlike ODEs based on pixel-based convolutional ﬁlters, the combination with structured ﬁlters (SRFs) also provide an intuitive way to parametrize the evolution of the kernels as a function of depth.
Revised: Unlike ODEs based on pixel-based convolutional ﬁlters, the combination with structured ﬁlters (SRFs) also provide an intuitive way to parametrize the evolution of the kernels as a function of depth.

Original: Ultimately, we see spatio-temporally continuous representations in end-to-end trainable networks as a link between modern CNN architectures and computational models of biological vision.
Revised: Ultimately, we see spatio-temporally continuous representations in end-to-end trainable networks as a link between modern CNN architectures and computational models of biological vision.

Original: Although for small spatio-temporal scales it may be more appropriate to use discrete descriptions of biological neurons, such as photoreceptors or populations located in spatially discrete locations, or considering temporally distinct spiking dynamics, computational models using continuous population activity or rate-based models provide reasonably good explanations of phenomena observed at the network or systems level (Ben-Yishai et al., 1995; Dayan & Abbott, 2001).
Revised: Although for small spatio-temporal scales it may be more appropriate to use discrete descriptions of biological neurons, such as photoreceptors or populations located in spatially discrete locations, or to consider temporally distinct spiking dynamics, computational models using continuous population activity or rate-based models provide reasonably good explanations of phenomena observed at the network or systems level (Ben-Yishai et al., 1995; Dayan & Abbott, 2001).

Original: We believe such larger scale computational models align well with the purposes of computer vision.
Revised: We believe such larger scale computational models align well with the purposes of computer vision.

Original: 2. S TRUCTURED RECEPTIVE FIELDS
Revised: 2. S TRUCTURED RECEPTIVE FIELDS

Original: We use the multiscale local N-jet formulation (Florack et al., 1996) to deﬁne the ﬁlters in convolutional layers.
Revised: We use the multiscale local N-jet formulation (Florack et al., 1996) to deﬁne the ﬁlters in convolutional layers.

Original: Structured receptive ﬁelds (SRFs) based on the Gaussian N-jet basis functions are highly applicable to CNNs, as they represent a Taylor expansion of the input image or feature maps in a local neighbourhood in space and scale, and can be used to approximate pixel-based ﬁlters (Appendix A.1).
Revised: Structured receptive ﬁelds (SRFs) based on the Gaussian N-jet basis functions are highly applicable to CNNs, as they represent a Taylor expansion of the input image or feature maps in a local neighbourhood in space and scale, and can be used to approximate pixel-based ﬁlters (Appendix A.1).

Original: This means that each ﬁlter F ( x, y ; σ ) in the network is a weighted sum of N basis functions, which are partial derivatives of the isotropic two-dimensional Gaussian function
Revised: This means that each ﬁlter F ( x, y ; σ ) in the network is a weighted sum of N basis functions, which are partial derivatives of the isotropic two-dimensional Gaussian function

Original: G ( x, y ; σ ) =
Revised: G ( x, y ; σ ) =

Original: [Equation]
Revised: [Equation]

Original: The scale, or the spatial extent, of the ﬁlter is explicitly modelled in the σ parameter of the Gaussian, which also indirectly determines the spatial frequency response of the SRF (Figure 1).
Revised: The scale, or the spatial extent, of the ﬁlter is explicitly modelled in the σ parameter of the Gaussian, which also indirectly determines the spatial frequency response of the SRF (Figure 1).

Original: Formally, the N-jet formulation of an SRF ﬁlter F ( x, y ) is given by:
Revised: Formally, the N-jet formulation of an SRF ﬁlter F ( x, y ) is given by:

Original: [Equation]
Revised: [Equation]

Original: where G i,j ( x, y ; σ ) are the partial derivatives of the Gaussian G ( x, y ; σ ) with respect to x and y , N is the degree of the Taylor polynomial which determines the basis order, and α encodes the expansion coefﬁcients.
Revised: where G i,j ( x, y ; σ ) are the partial derivatives of the Gaussian G ( x, y ; σ ) with respect to x and y , N is the degree of the Taylor polynomial which determines the basis order, and α encodes the expansion coefﬁcients.

Original: Deﬁned this way, SRFs have favourable properties over pixel-based ﬁlters.
Revised: Deﬁned this way, SRFs have favourable properties over pixel-based ﬁlters.

Original: SRF ﬁlters are steerable by the coefﬁcients α and the basis functions are spatially separable.
Revised: SRF ﬁlters are steerable by the coefﬁcients α and the basis functions are spatially separable.

Original: Likewise, due to their spatially continuous description, the ﬁlters can be trivially scaled, or rotated, without interpolation.
Revised: Likewise, due to their spatially continuous description, the ﬁlters can be trivially scaled, or rotated, without interpolation.

Original: Figure 1 shows the N-jet approximation of two ﬁlters in two different scales σ 1 and σ 2 .
Revised: Figure 1 shows the N-jet approximation of two ﬁlters in two different scales σ 1 and σ 2 .

Original: We note that both the coefﬁcients α and the scale σ are learnable ﬁlter parameters.
Revised: We note that both the coefﬁcients α and the scale σ are learnable ﬁlter parameters.

Original: Instead of ﬁxing the scale σ a priori and optimizing for α as in Jacobsen et al.
Revised: Instead of ﬁxing the scale σ a priori and optimizing for α as in Jacobsen et al.

Original: (2016) and Sosnovik et al.
Revised: (2016) and Sosnovik et al.

Original: (2020), we integrate both these parameters in the network optimization, thus learning not only the shape but also the spatial support of the ﬁlters.
Revised: (2020), we integrate both these parameters in the network optimization, thus learning not only the shape but also the spatial support of the ﬁlters.

Original: 2. N EURAL ODE S
Revised: 2. N EURAL ODE S

Original: We model the continuous evolution of feature maps within an ‘ODE block’.
Revised: We model the continuous evolution of feature maps within an ‘ODE block’.

Original: Formally, an ODE block contains a stack of M convolutional layers, each with its own convolutional ﬁlters w m with m = 1 . . .
Revised: Formally, an ODE block contains a stack of M convolutional layers, each with its own convolutional ﬁlters w m with m = 1 . . .

Original: M , followed by normalization G norm ( · ) and non-linear activation CELU ( · ) functions.
Revised: M , followed by normalization G norm ( · ) and non-linear activation CELU ( · ) functions.

Original: Following the notations of Chen et al.
Revised: Following the notations of Chen et al.

Original: and Ruthotto & Haber (2019), we deﬁne the equations of motion for the feature states h ∈ R n as:
Revised: and Ruthotto & Haber (2019), we deﬁne the equations of motion for the feature states h ∈ R n as:

Original: [Equation]
Revised: [Equation]

Original: where g ( x ) =
Revised: where g ( x ) =

Original: CELU ( G norm ( x )) , the linear operators K m ∈ R n × n denote the convolution operators parametrized by w m , and J m ∈ R n × R denote the linear explicit t -terms parametrized by d m .
Revised: CELU ( G norm ( x )) , the linear operators K m ∈ R n × n denote the convolution operators parametrized by w m , and J m ∈ R n × R denote the linear explicit t -terms parametrized by d m .

Original: The ﬁlters w m , d m ( θ ) are functions of some learnable parameters θ .
Revised: The ﬁlters w m , d m ( θ ) are functions of some learnable parameters θ .

Original: As a typical CNN convolution operator, K m ( w m ) with 2 input and 2 output channels can be written as
Revised: As a typical CNN convolution operator, K m ( w m ) with 2 input and 2 output channels can be written as

Original: [Equation]
Revised: [Equation]

Original: with w mji the convolutional kernels for input channel i and output channel j of the m -th convolution.
Revised: with w mji the convolutional kernels for input channel i and output channel j of the m -th convolution.

Original: In this deﬁnition, the discrete depth of feed-forward networks such as ResNets is reimagined as a continuous dimension deﬁned by time t , where the input image deﬁnes the initial conditions h (0) .
Revised: In this deﬁnition, the discrete depth of feed-forward networks such as ResNets is reimagined as a continuous dimension deﬁned by time t , where the input image deﬁnes the initial conditions h (0) .

Original: For the rest of this paper, we stick to the original ODE-Net interpretation that the number of function evaluations performed by the numerical ODE solver is analogous to network depth.
Revised: For the rest of this paper, we stick to the original ODE-Net interpretation that the number of function evaluations performed by the numerical ODE solver is analogous to network depth.

Original: In accordance with conventional ResNet blocks, we pick M = 2 .
Revised: In accordance with conventional ResNet blocks, we pick M = 2 .

Original: Based on the original implementation by Chen et al.
Revised: Based on the original implementation by Chen et al.

Original: (2018), G norm is deﬁned as group normalization (Wu & He, 2018).
Revised: (2018), G norm is deﬁned as group normalization (Wu & He, 2018).

Original: For generalized compatibility with neural ODEs and the adjoint method, we choose a non-linear activation function with a theoretically unique and bounded adjoint, namely continuously differentiable exponential linear units, or CELU (Barron, 2017).
Revised: For generalized compatibility with neural ODEs and the adjoint method, we choose a non-linear activation function with a theoretically unique and bounded adjoint, namely continuously differentiable exponential linear units, or CELU (Barron, 2017).

Original: Similarly, we keep the linear dependence of the equations of motion on time (or network depth) t .
Revised: Similarly, we keep the linear dependence of the equations of motion on time (or network depth) t .

Original: Finally, we adapt the GPU implementation of
Revised: Finally, we adapt the GPU implementation of

Original: ODE solvers 1 to solve the equations of motion for a predeﬁned time interval t ∈ [0 , T ] using the adaptive step size DOPRI method.
Revised: ODE solvers 1 to solve the equations of motion for a predeﬁned time interval t ∈ [0 , T ] using the adaptive step size DOPRI method.

Original: 2. D EEP C ONTINUOUS N ETWORKS WITH SRF S AND NEURAL ODE S
Revised: 2. D EEP C ONTINUOUS N ETWORKS WITH SRF S AND NEURAL ODE S

Original: We formulate deep continuous networks (DCNs) by employing learnable, continuous SRF ﬁlter descriptions to deﬁne the weights in the evolution of a neural ODE.
Revised: We formulate deep continuous networks (DCNs) by employing learnable, continuous SRF ﬁlter descriptions to deﬁne the weights in the evolution of a neural ODE.

Original: This means that for DCNs, each w mji in Eq.
Revised: This means that for DCNs, each w mji in Eq.

Original: 2 is a discretization of the continuous SRF ﬁlter F α ( x, y ; σ ) given in Eq.
Revised: 2 is a discretization of the continuous SRF ﬁlter F α ( x, y ; σ ) given in Eq.

Original: 1, sampled in [ − 2 σ, 2 σ ] .
Revised: 1, sampled in [ − 2 σ, 2 σ ] .

Original: We deﬁne the spatial axes such that the width and height of a pixel is δ x = δ y
Revised: We deﬁne the spatial axes such that the width and height of a pixel is δ x = δ y

Original: = 

Revised: =

Original: and the sampling rate is 1 ( δ − 1 x ).
Revised: and the sampling rate is 1 ( δ − 1 x ).

Original: α mji and σ m are trainable ﬁlter parameters, where σ mji = σ m is shared between the ﬁlters in a convolutional layer unless stated otherwise.
Revised: α mji and σ m are trainable ﬁlter parameters, where σ mji = σ m is shared between the ﬁlters in a convolutional layer unless stated otherwise.

Original: All our code is available at 2 .
Revised: All our code is available at 2 .

Original: Network architecture and training.
Revised: Network architecture and training.

Original: We construct DCNs by stacking ODE blocks separated by downsampling blocks (Figure 2).
Revised: We construct DCNs by stacking ODE blocks separated by downsampling blocks (Figure 2).

Original: Each downsampling block is a sequence of normalization, nonlinear activation and strided convolution.
Revised: Each downsampling block is a sequence of normalization, nonlinear activation and strided convolution.

Original: We use a convolutional layer for increasing the channel dimensionality at the input level and employ global average pooling and a fully connected layer at the output level.
Revised: We use a convolutional layer for increasing the channel dimensionality at the input level and employ global average pooling and a fully connected layer at the output level.

Original: We train our networks using cross entropy loss and the CIFAR-10 dataset (Krizhevsky, 2009).
Revised: We train our networks using cross entropy loss and the CIFAR-10 dataset (Krizhevsky, 2009).

Original: (See Appendices A.2-A.3 for further details regarding training parameters.)
Revised: (See Appendices A.2-A.3 for further details regarding training parameters.)

Original: As a baseline without spatial continuity, we compare DCN performance to the ODE-Net introduced in Chen et al.
Revised: As a baseline without spatial continuity, we compare DCN performance to the ODE-Net introduced in Chen et al.

Original: (2018), where the convolutions within the ODE blocks are performed using discrete, pixel-based kernels, with k × k parameters.
Revised: (2018), where the convolutions within the ODE blocks are performed using discrete, pixel-based kernels, with k × k parameters.

Original: As a baseline without temporal continuity, we deﬁne the ‘ResNet-blocks’ model where the ODE blocks are replaced by generic, discrete ResNet blocks, comprising two convolutional layers and a skip connection, with comparable number of parameters to the ODE-Net.
Revised: As a baseline without temporal continuity, we deﬁne the ‘ResNet-blocks’ model where the ODE blocks are replaced by generic, discrete ResNet blocks, comprising two convolutional layers and a skip connection, with comparable number of parameters to the ODE-Net.

Original: This is similar to the baseline model used in Chen et al.
Revised: This is similar to the baseline model used in Chen et al.

Original: In the ResNet-SRFblocks model, we provide the discrete-time and continuous-space baseline by replacing the k × k ﬁlter deﬁnition of ResNet-blocks with SRF deﬁnitions.
Revised: In the ResNet-SRFblocks model, we provide the discrete-time and continuous-space baseline by replacing the k × k ﬁlter deﬁnition of ResNet-blocks with SRF deﬁnitions.

Original: Input Image (32x32x3) k x k conv,
Revised: Input Image (32x32x3) k x k conv,

Original: ODE Block 1,
Revised: ODE Block 1,

Original: Downsampling block 1,
Revised: Downsampling block 1,

Original: ODE Block 2,
Revised: ODE Block 2,

Original: ODE Block 3,fc,normalization nonlinear activation
Revised: ODE Block 3,fc,normalization nonlinear activation

Original: Global Average Pooling
Revised: Global Average Pooling

Original: [Figure]
Revised: [Figure]

Original: The convolutional kernel size k depends on the model, and is learned during training in DCNs.
Revised: The convolutional kernel size k depends on the model, and is learned during training in DCNs.

Original: The equations of motion (Eq. 2) are solved within
Revised: The equations of motion (Eq. 2) are solved within

Original: ODE blocks.
Revised: ODE blocks.

Original: We test two versions of DCNs and ResNet-SRF-blocks to quantify the viability of SRF ﬁlters outside of the ODE blocks.
Revised: We test two versions of DCNs and ResNet-SRF-blocks to quantify the viability of SRF ﬁlters outside of the ODE blocks.

Original: In DCN-ODE and ResNet-SRF-blocks we use the SRF ﬁlters only within the ODE (ResNet) blocks, and for the remaining layers we use discrete kernels with the same hyperparameters as the baselines.
Revised: In DCN-ODE and ResNet-SRF-blocks we use the SRF ﬁlters only within the ODE (ResNet) blocks, and for the remaining layers we use discrete kernels with the same hyperparameters as the baselines.

Original: In the second version,
Revised: In the second version,

Original: DCN-full (ResNet-SRF-full)
Revised: DCN-full (ResNet-SRF-full)

Original: , we use spatially continuous kernels everywhere, including the downsampling layers.
Revised: , we use spatially continuous kernels everywhere, including the downsampling layers.

Original: Finally, we investigate the case where we drop scale sharing within a layer, and optimize the scale parameter σ mji independently for each input channel i and output channel j , which we call DCN σ ji .
Revised: Finally, we investigate the case where we drop scale sharing within a layer, and optimize the scale parameter σ mji independently for each input channel i and output channel j , which we call DCN σ ji .

Original: Meta-parametrization.
Revised: Meta-parametrization.

Original: DCNs enable us to parametrize the trainable ﬁlter parameters α and σ as a function of time t .
Revised: DCNs enable us to parametrize the trainable ﬁlter parameters α and σ as a function of time t .

Original: This both enables the kernels to vary smoothly over time, and lets us deﬁne temporal dynamics for the neuronal responses in our network.
Revised: This both enables the kernels to vary smoothly over time, and lets us deﬁne temporal dynamics for the neuronal responses in our network.

Original: We test the viability of such models by introducing DCN variants where σ and/or α are deﬁned using linear or quadratic functions of t and learnable parameters a , b , c , a s , b s , a α and b α (Table 3).
Revised: We test the viability of such models by introducing DCN variants where σ and/or α are deﬁned using linear or quadratic functions of t and learnable parameters a , b , c , a s , b s , a α and b α (Table 3).

Original:  E XPERIMENTAL ANALYSIS 3. P ARAMETER REDUCTION AND DATA EFFICIENCY
Revised:  E XPERIMENTAL ANALYSIS 3. P ARAMETER REDUCTION AND DATA EFFICIENCY

Original: Similar to biological models, where analytical receptive ﬁelds limit the scope of the model using prior information, we ﬁnd that DCNs are more parameter efﬁcient compared to baseline networks.
Revised: Similar to biological models, where analytical receptive ﬁelds limit the scope of the model using prior information, we ﬁnd that DCNs are more parameter efﬁcient compared to baseline networks.

Original: Evaluated on
Revised: Evaluated on

Original: CIFAR-10, DCNs perform on par with baselines, despite using SRFs of a small basis order 2, which means each ﬁlter shape is deﬁned by only 6 free parameters as opposed to 9 for the conventional 3 × 3 kernels (Table 1).
Revised: CIFAR-10, DCNs perform on par with baselines, despite using SRFs of a small basis order 2, which means each ﬁlter shape is deﬁned by only 6 free parameters as opposed to 9 for the conventional 3 × 3 kernels (Table 1).

Original: In addition, we ﬁnd that parameter reduction via the use of SRFs with a small basis order also leads to data efﬁciency.
Revised: In addition, we ﬁnd that parameter reduction via the use of SRFs with a small basis order also leads to data efﬁciency.

Original: When trained on a subset of CIFAR-10 images (small-data regime), DCNs outperform the discrete baseline networks (Table 2a)
Revised: When trained on a subset of CIFAR-10 images (small-data regime), DCNs outperform the discrete baseline networks (Table 2a)

Original: [Table]
Revised: [Table]

Original: As an additional demonstration of the versatility of the spatio-temporal representations learned by
Revised: As an additional demonstration of the versatility of the spatio-temporal representations learned by

Original: DCNs
Revised: DCNs

Original: [Table]
Revised: [Table]

Original: we perform an image reconstruction experiment.
Revised: we perform an image reconstruction experiment.

Original: We use the feature maps generated by the networks trained on the CIFAR-10 classiﬁcation task (output of ODE Block 3 in Fig. 2)
Revised: We use the feature maps generated by the networks trained on the CIFAR-10 classiﬁcation task (output of ODE Block 3 in Fig. 2)

Original: [Table]
Revised: [Table]

Original: as input to a decoder network.
Revised: as input to a decoder network.

Original: The decoder networks are composed of 2 DCN-ODE, ODE-Net or ResNet blocks, separated by bilinear upscaling layers and 1 × 1 convolutions to reduce the output dimensionality
Revised: The decoder networks are composed of 2 DCN-ODE, ODE-Net or ResNet blocks, separated by bilinear upscaling layers and 1 × 1 convolutions to reduce the output dimensionality

Original: [Table] [Table]
Revised: [Table] [Table]

Original: to three (RGB).
Revised: to three (RGB).

Original: We train the networks to reconstruct CIFAR-10 images using mean squared error (MSE) loss.
Revised: We train the networks to reconstruct CIFAR-10 images using mean squared error (MSE) loss.

Original: We ﬁnd that the DCN models outperform discrete baseline models on the validation set (Table 2b), despite having a lower number of parameters as before.
Revised: We ﬁnd that the DCN models outperform discrete baseline models on the validation set (Table 2b), despite having a lower number of parameters as before.

Original: Additional details and example images are shown in Appendix A.4.
Revised: Additional details and example images are shown in Appendix A.4.

Original: Similarly, we ﬁnd that meta-parametrized DCN variants match the classiﬁcation performance of baselines and may outperform DCNs with static weights (Table 4).
Revised: Similarly, we ﬁnd that meta-parametrized DCN variants match the classiﬁcation performance of baselines and may outperform DCNs with static weights (Table 4).

Original: This is an interesting ﬁnding since we test only a few models, with little hyperparameter optimization, which indicates that DCNs can potentially be used to parametrize the dependence of convolutional kernel weights on network depth, for further parameter reduction.
Revised: This is an interesting ﬁnding since we test only a few models, with little hyperparameter optimization, which indicates that DCNs can potentially be used to parametrize the dependence of convolutional kernel weights on network depth, for further parameter reduction.

Original: 3. L INK WITH BIOLOGICAL MODELS
Revised: 3. L INK WITH BIOLOGICAL MODELS

Original: Scale ﬁtting.
Revised: Scale ﬁtting.

Original: As an advantage over conventional CNNs, it is possible to directly investigate the optimal receptive ﬁeld (RF) size in each DCN block after training, since DCNs ﬁt the kernel scale σ explicitly.
Revised: As an advantage over conventional CNNs, it is possible to directly investigate the optimal receptive ﬁeld (RF) size in each DCN block after training, since DCNs ﬁt the kernel scale σ explicitly.

Original: We observe an upward trend in the SRF scale σ with the depth of the convolutional layer within the network (Fig. 3a).
Revised: We observe an upward trend in the SRF scale σ with the depth of the convolutional layer within the network (Fig. 3a).

Original: While the RF size grows with depth also in conventional CNNs, it typically grows linearly over convolutions, as the kernel size is constant across layers.
Revised: While the RF size grows with depth also in conventional CNNs, it typically grows linearly over convolutions, as the kernel size is constant across layers.

Original: This is a limitation in most CNNs that the visual system does not necessarily have.
Revised: This is a limitation in most CNNs that the visual system does not necessarily have.

Original: DCNs, on the other hand, can learn RF sizes which grow non-linearly as a function of depth, which seems to be in line with the behaviour in downstream visual areas (Smith et al., 2001).
Revised: DCNs, on the other hand, can learn RF sizes which grow non-linearly as a function of depth, which seems to be in line with the behaviour in downstream visual areas (Smith et al., 2001).

Original: In addition, we plot the distribution of learned σ ji in different ODE blocks of the model DCN σ ji (Fig. 3b).
Revised: In addition, we plot the distribution of learned σ ji in different ODE blocks of the model DCN σ ji (Fig. 3b).

Original: Note that the scale parameter σ controls the bandwidth of the SRF ﬁlters and is thus related to their spatial frequency response.
Revised: Note that the scale parameter σ controls the bandwidth of the SRF ﬁlters and is thus related to their spatial frequency response.

Original: We ﬁnd that the σ ji distributions after training are approximately log-normal and display a positive skew, which is consistent with the scale and spatial frequency tuning distributions in the primate visual system (Yu et al., 2010).
Revised: We ﬁnd that the σ ji distributions after training are approximately log-normal and display a positive skew, which is consistent with the scale and spatial frequency tuning distributions in the primate visual system (Yu et al., 2010).

Original: We believe these results are promising for bridging the gap between deep learning and traditional models of biological systems.
Revised: We believe these results are promising for bridging the gap between deep learning and traditional models of biological systems.

Original: Pattern completion.
Revised: Pattern completion.

Original: Established models from computational neuroscience, with continuous temporal dynamics and well-deﬁned recurrent interaction structures, such as the Ermentrout-Cowan model (Bressloff et al., 2001), or neural ﬁeld models (Amari, 1977), display interesting high-level phenomena such as spontaneous pattern formation and travelling waves (Coombes, 2005).
Revised: Established models from computational neuroscience, with continuous temporal dynamics and well-deﬁned recurrent interaction structures, such as the Ermentrout-Cowan model (Bressloff et al., 2001), or neural ﬁeld models (Amari, 1977), display interesting high-level phenomena such as spontaneous pattern formation and travelling waves (Coombes, 2005).

Original: Such models employ local, distance-dependent interactions, similar to the SRF based ODE blocks in the
Revised: Such models employ local, distance-dependent interactions, similar to the SRF based ODE blocks in the

Original: DCN formulation.
Revised: DCN formulation.

Original: Based on this resemblance, we hypothesize that DCNs can perform well in the case of locally missing information in images, through pattern completion at the feature map level.
Revised: Based on this resemblance, we hypothesize that DCNs can perform well in the case of locally missing information in images, through pattern completion at the feature map level.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: l
Revised: l

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original: 1 : c
Revised: 1 : c

Original: [Figure]
Revised: [Figure]

Original: We test this hypothesis by masking n × n pixels on the CIFAR-10 validation set at test time.
Revised: We test this hypothesis by masking n × n pixels on the CIFAR-10 validation set at test time.

Original: The masks have zero pixel values, and are placed at the center of the image.
Revised: The masks have zero pixel values, and are placed at the center of the image.

Original: We ﬁnd that when confronted with a small patch of missing information, DCNs can generate feature maps similar to those obtained from intact images.
Revised: We ﬁnd that when confronted with a small patch of missing information, DCNs can generate feature maps similar to those obtained from intact images.

Original: Speciﬁcally, we observe that the difference D ( t ) =
Revised: Speciﬁcally, we observe that the difference D ( t ) =

Original: 1 A (cid:80) | h
Revised: 1 A (cid:80) | h

Original: im ( t ) − h
Revised: im ( t ) − h

Original: im masked ( t ) | between the feature maps generated by an intact image
Revised: im masked ( t ) | between the feature maps generated by an intact image

Original: h
Revised: h

Original: im ( t ) and a masked image
Revised: im ( t ) and a masked image

Original: h
Revised: h

Original: im masked ( t ) , normalized by the amplitude of the intact image A , is reduced within an ODE block (Figure 4a).
Revised: im masked ( t ) , normalized by the amplitude of the intact image A , is reduced within an ODE block (Figure 4a).

Original: In terms of the overall classiﬁcation performance with masked images at test time, we ﬁnd that DCNs are marginally more robust against zero-masking than baselines (Fig. 3c).
Revised: In terms of the overall classiﬁcation performance with masked images at test time, we ﬁnd that DCNs are marginally more robust against zero-masking than baselines (Fig. 3c).

Original: 3. C ONTRAST ROBUSTNESS AND COMPUTATIONAL EFFICIENCY
Revised: 3. C ONTRAST ROBUSTNESS AND COMPUTATIONAL EFFICIENCY

Original: The selectivity of neuronal responses is invariant to contrast in mammalian vision (Sclar & Freeman, 1982; Skottun et al., 1987).
Revised: The selectivity of neuronal responses is invariant to contrast in mammalian vision (Sclar & Freeman, 1982; Skottun et al., 1987).

Original: However, we observe that DCN and ODE-Net models are sensitive to changes in input contrast.
Revised: However, we observe that DCN and ODE-Net models are sensitive to changes in input contrast.

Original: This is not unexpected since ODE blocks compute the solution to the initial value problem posed by the equations of motion and the input feature maps, and contrast variations change the initial conditions.
Revised: This is not unexpected since ODE blocks compute the solution to the initial value problem posed by the equations of motion and the input feature maps, and contrast variations change the initial conditions.

Original: To quantify this sensitivity we vary the contrast c of the input images at test time, where for each image H i in the CIFAR-10 validation set we deﬁne the input as H j = cH i .
Revised: To quantify this sensitivity we vary the contrast c of the input images at test time, where for each image H i in the CIFAR-10 validation set we deﬁne the input as H j = cH i .

Original: When naively changing the input contrast c this way, we ﬁnd that the validation accuracy decays rapidly for both models (solid lines in Fig.
Revised: When naively changing the input contrast c this way, we ﬁnd that the validation accuracy decays rapidly for both models (solid lines in Fig.

Original: 4b, top).
Revised: 4b, top).

Original: Empirically, we notice that, with the appropriate choice of normalization functions, the input contrast c has a direct effect on the time scales of the solution h ( t ) .
Revised: Empirically, we notice that, with the appropriate choice of normalization functions, the input contrast c has a direct effect on the time scales of the solution h ( t ) .

Original: Based on this observation, we heuristically test whether scaling the integration time interval T (used during training) of ODE block 1 by c for each input image j , T j = c T , can improve contrast robustness at test time.
Revised: Based on this observation, we heuristically test whether scaling the integration time interval T (used during training) of ODE block 1 by c for each input image j , T j = c T , can improve contrast robustness at test time.

Original: We ﬁnd that with the scaled integration interval DCN validation accuracy is relatively robust against changes in contrast c , compared to naive baselines and ODE-Net, until c << 1 when time scales become too fast and the ODE solver becomes unstable for all models (dashed lines in Fig.
Revised: We ﬁnd that with the scaled integration interval DCN validation accuracy is relatively robust against changes in contrast c , compared to naive baselines and ODE-Net, until c << 1 when time scales become too fast and the ODE solver becomes unstable for all models (dashed lines in Fig.

Original: 4b, top).
Revised: 4b, top).

Original: Interestingly, we observe a reduction in the number of function evaluations (NFEs) in ODE blockfor c < 1 (Fig. 4b, middle).
Revised: Interestingly, we observe a reduction in the number of function evaluations (NFEs) in ODE blockfor c < 1 (Fig. 4b, middle).

Original: Furthermore, we show that as long as the error tolerance of the ODE blocks are not decreased, this effect can be exploited by scaling the input feature maps of all ODE blocks by c for signiﬁcant computational savings.
Revised: Furthermore, we show that as long as the error tolerance of the ODE blocks are not decreased, this effect can be exploited by scaling the input feature maps of all ODE blocks by c for signiﬁcant computational savings.

Original: We ﬁnd that decreasing c leads to considerable efﬁciency improvements, where total NFEs can be reduced from 102 to 60, (for c = 1 and 0.06), with less than 0.5% loss in accuracy (Fig. 4b, bottom).
Revised: We ﬁnd that decreasing c leads to considerable efﬁciency improvements, where total NFEs can be reduced from 102 to 60, (for c = 1 and 0.06), with less than 0.5% loss in accuracy (Fig. 4b, bottom).

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Spatially continuous ﬁlter representations.
Revised: Spatially continuous ﬁlter representations.

Original: Structured ﬁlters have been traditionally used in computer vision for extracting image structure at multiple scales.
Revised: Structured ﬁlters have been traditionally used in computer vision for extracting image structure at multiple scales.

Original: N-jet ﬁlter basis is ﬁrst introduced by Florack et al.
Revised: N-jet ﬁlter basis is ﬁrst introduced by Florack et al.

Original: based on previous work on Gaussian scale-spaces (Florack et al., 1992; Lindeberg, 2013).
Revised: based on previous work on Gaussian scale-spaces (Florack et al., 1992; Lindeberg, 2013).

Original: We use the N-jet basis in order to approximate convolutional ﬁlters, as they enable a spatially continuous representation with an explicit scale parameter σ .
Revised: We use the N-jet basis in order to approximate convolutional ﬁlters, as they enable a spatially continuous representation with an explicit scale parameter σ .

Original: [Figure]
Revised: [Figure]

Original: Similar to the N-jet basis, a set of oriented multi-scale wavelets, called a steerable pyramid, is proposed by Simoncelli et al. (1992) and complex wavelets have been used by Bruna & Mallat (2013) and Mallat (2012) as part of scattering transforms.
Revised: Similar to the N-jet basis, a set of oriented multi-scale wavelets, called a steerable pyramid, is proposed by Simoncelli et al. (1992) and complex wavelets have been used by Bruna & Mallat (2013) and Mallat (2012) as part of scattering transforms.

Original: CNN ﬁlters based on linear combinations of Gabor wavelets are adopted by Luan et al.
Revised: CNN ﬁlters based on linear combinations of Gabor wavelets are adopted by Luan et al.

Original: (2017) while Worrall et al.
Revised: (2017) while Worrall et al.

Original: (2017) propose circular harmonics, as spatially continuous ﬁlter representations.
Revised: (2017) propose circular harmonics, as spatially continuous ﬁlter representations.

Original: Similar to our approach, Shelhamer et al.
Revised: Similar to our approach, Shelhamer et al.

Original: (2019) combine free-form ﬁlters with Gaussian kernels, thus learning the ﬁlter resolution.
Revised: (2019) combine free-form ﬁlters with Gaussian kernels, thus learning the ﬁlter resolution.

Original: Likewise, Xiong et al.
Revised: Likewise, Xiong et al.

Original: (2020) learn ﬁlter sizes using Gaussian kernels optimized using variational inference.
Revised: (2020) learn ﬁlter sizes using Gaussian kernels optimized using variational inference.

Original: Here, we use the N-jet framework based on Gaussian derivatives as in Jacobsen et al.
Revised: Here, we use the N-jet framework based on Gaussian derivatives as in Jacobsen et al.

Original: (2016), however our main motivation is retaining compatibility with biological models.
Revised: (2016), however our main motivation is retaining compatibility with biological models.

Original: Also, unlike Jacobsen et al.
Revised: Also, unlike Jacobsen et al.

Original: we learn the scale parameter σ during training.
Revised: we learn the scale parameter σ during training.

Original: Continuous time representations in deep networks.
Revised: Continuous time representations in deep networks.

Original: Along with work by Lu et al.
Revised: Along with work by Lu et al.

Original: and Ruthotto & Haber (2019), networks continuous in the temporal (or depth) dimension have been proposed by Chen et al.
Revised: and Ruthotto & Haber (2019), networks continuous in the temporal (or depth) dimension have been proposed by Chen et al.

Original: (2018) under the name neural ordinary differential equations (ODEs).
Revised: (2018) under the name neural ordinary differential equations (ODEs).

Original: They propose ODE-Nets based on the ResNet formulation (He et al., 2016) for classiﬁcation tasks, which is used as a baseline in this paper.
Revised: They propose ODE-Nets based on the ResNet formulation (He et al., 2016) for classiﬁcation tasks, which is used as a baseline in this paper.

Original: In this work we focus mainly on an image classiﬁcation task, however, there is extensive ongoing work on generative models and normalizing ﬂows using the neural
Revised: In this work we focus mainly on an image classiﬁcation task, however, there is extensive ongoing work on generative models and normalizing ﬂows using the neural

Original: ODE continuous time interpretation (Salman et al., 2018; Grathwohl et al., 2019).
Revised: ODE continuous time interpretation (Salman et al., 2018; Grathwohl et al., 2019).

Original: We note that
Revised: We note that

Original: DCNs can be readily incorporated into continuous ﬂow models, as well as other spatio-temporally continuous DNN interpretations based on partial differential equations (Ruthotto & Haber, 2019).
Revised: DCNs can be readily incorporated into continuous ﬂow models, as well as other spatio-temporally continuous DNN interpretations based on partial differential equations (Ruthotto & Haber, 2019).

Original: Even though the adjoint method described in Chen et al.
Revised: Even though the adjoint method described in Chen et al.

Original: (2018) offers considerable computational savings, especially in terms of memory, recent work has improved upon it both in terms of stability and computational efﬁciency  (Dupont et al., 2019; Finlay et al., 2020).
Revised: (2018) offers considerable computational savings, especially in terms of memory, recent work has improved upon it both in terms of stability, computational efﬁciency and performance (Dupont et al., 2019; Finlay et al., 2020; Zhuang et al., 2020b).

Original: Likewise, the contrast robust formulation of DCNs, as well as the synergy between the O (1) memory complexity of the adjoint method and spatially separable SRF ﬁlters (the implementations of which may otherwise inﬂate the memory cost) provide potential computational beneﬁts.
Revised: Likewise, the contrast robust formulation of DCNs, as well as the synergy between the O (1) memory complexity of the adjoint method and spatially separable SRF ﬁlters (the implementations of which may otherwise inﬂate the memory cost) provide potential computational beneﬁts.

Original: Other studies have suggested that, similar to our DCN variants where the ﬁlter deﬁnitions are independent of time, neural ODEs based on ResNet architectures with weight sharing can be interpreted as recurrent neural networks (Rousseau et al., 2019;
Revised: Other studies have suggested that, similar to our DCN variants where the ﬁlter deﬁnitions are independent of time, neural ODEs based on ResNet architectures with weight sharing can be interpreted as recurrent neural networks (Rousseau et al., 2019;

Original: Kim et al., 2016), which bridges the gap between deep learning, dynamical systems and the primate visual cortex (Massaroli et al., 2020; Liao & Poggio, 2016).
Revised: Kim et al., 2016), which bridges the gap between deep learning, dynamical systems and the primate visual cortex (Massaroli et al., 2020; Liao & Poggio, 2016).

Original: Similar to these works, we capitalize on the parallels between neural ODEs and the dynamical systems approach of the computational models of recurrent, biological circuits in order
Revised: Similar to these works, we capitalize on the parallels between neural ODEs and the dynamical systems approach of the computational models of recurrent, biological circuits in order

Original: CNNs and RNNs as models of biological networks.
Revised: CNNs and RNNs as models of biological networks.

Original: There is extensive prior work on CNNs and recurrent neural networks (RNNs) for modeling biological computation.
Revised: There is extensive prior work on CNNs and recurrent neural networks (RNNs) for modeling biological computation.

Original: The visual cortex is highly recurrent (Dayan & Abbott, 2001; Liao & Poggio, 2016) which is thought to be responsible for complex neuronal dynamics (Ben-Yishai et al., 1995; Angelucci & Bressloff, 2006).
Revised: The visual cortex is highly recurrent (Dayan & Abbott, 2001; Liao & Poggio, 2016) which is thought to be responsible for complex neuronal dynamics (Ben-Yishai et al., 1995; Angelucci & Bressloff, 2006).

Original: Accordingly, computational models with lateral connections (Sompolinsky et al., 1988; Ernst et al., 2001) and more recently RNNs (Laje & Buonomano, 2013; Mante et al., 2013; Mastrogiuseppe & Ostojic, 2018) have been extensively used as models of biological neural computation.
Revised: Accordingly, computational models with lateral connections (Sompolinsky et al., 1988; Ernst et al., 2001) and more recently RNNs (Laje & Buonomano, 2013; Mante et al., 2013; Mastrogiuseppe & Ostojic, 2018) have been extensively used as models of biological neural computation.

Original: For example the ﬁrstorder reduced and controlled error (FORCE) algorithm, have been used to reproduce the dynamics of different biological circuits (Sussillo & Abbott, 2009; Laje & Buonomano, 2013;
Revised: For example the ﬁrstorder reduced and controlled error (FORCE) algorithm, have been used to reproduce the dynamics of different biological circuits (Sussillo & Abbott, 2009; Laje & Buonomano, 2013;

Original: Carnevale et al., 2015; Rajan et al., 2016; Enel et al., 2016).
Revised: Carnevale et al., 2015; Rajan et al., 2016; Enel et al., 2016).

Original: Similarly, optimization via gradient-based algorithms such as the Hessian-free method (HF) or stochastic gradient-descent (SGD) have been adopted to train recurrent networks (Mante et al., 2013; Barak et al., 2013; Song et al., 2016) to replicate experimental observations.
Revised: Similarly, optimization via gradient-based algorithms such as the Hessian-free method (HF) or stochastic gradient-descent (SGD) have been adopted to train recurrent networks (Mante et al., 2013; Barak et al., 2013; Song et al., 2016) to replicate experimental observations.

Original: It has also been suggested to use spiking recurrent networks for improved physiological realism and efﬁciency (Kim & Chow, 2018;
Revised: It has also been suggested to use spiking recurrent networks for improved physiological realism and efﬁciency (Kim & Chow, 2018;

Original: Kim et al., 2019).
Revised: Kim et al., 2019).

Original: Finally, building on
Revised: Finally, building on

Original: CNN architectures, recurrent convolutional networks (RCNNs) have been proposed (Liang & Hu, 2015; Spoerer et al., 2017; Hu & Mihalas, 2018), which can emulate biological lateral connectivity structures and extra-classical receptive ﬁeld effects.
Revised: CNN architectures, recurrent convolutional networks (RCNNs) have been proposed (Liang & Hu, 2015; Spoerer et al., 2017; Hu & Mihalas, 2018), which can emulate biological lateral connectivity structures and extra-classical receptive ﬁeld effects.

Original: In contrast to typical RNNs, our model is based on the ResNet inspired model of neural ODEs, and in its current form (Eq. 2), does not accept time-variant input.
Revised: In contrast to typical RNNs, our model is based on the ResNet inspired model of neural ODEs, and in its current form (Eq. 2), does not accept time-variant input.

Original: Although with weight sharing
Revised: Although with weight sharing

Original: DCNs can be thought of as recurrent networks (Rousseau et al., 2019) and can be modiﬁed to process time-variant input (such as videos), in this paper we consider the DCN models to be an extension of conventional feed-forward CNNs, with extended temporal dynamics and continuous spatial representations, which are applicable to feed-forward models of the visual system similar to works by Lindsey et al.
Revised: DCNs can be thought of as recurrent networks (Rousseau et al., 2019) and can be modiﬁed to process time-variant input (such as videos), in this paper we consider the DCN models to be an extension of conventional feed-forward CNNs, with extended temporal dynamics and continuous spatial representations, which are applicable to feed-forward models of the visual system similar to works by Lindsey et al.

Original: ; Ecker et al.
Revised: ; Ecker et al.

Original: ; Schrimpf et al.
Revised: ; Schrimpf et al.

Original: ; Zhuang et al.
Revised: ; Zhuang et al.

Revised: (2020a).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We introduce DCNs, CNN models which learn spatio-temporally continuous representations, consistent with biological models.
Revised: We introduce DCNs, CNN models which learn spatio-temporally continuous representations, consistent with biological models.

Original: We showed that DCNs can match baseline performance in an image classiﬁcation task and outperform baselines in the small-data regime and in a reconstruction task, while using a smaller number of parameters.
Revised: We showed that DCNs can match baseline performance in an image classiﬁcation task and outperform baselines in the small-data regime and in a reconstruction task, while using a smaller number of parameters.

Original: Similarly, we have proposed different methods of metaparametrization of the convolutional ﬁlter as a function of time, which may not only be viable for network compression purposes, but also for modelling the temporal proﬁles of biological responses.
Revised: Similarly, we have proposed different methods of metaparametrization of the convolutional ﬁlter as a function of time, which may not only be viable for network compression purposes, but also for modelling the temporal proﬁles of biological responses.

Original: As a further link with biological models, we have demonstrated that the learned scale distributions in DCNs are compatible with experimental observations, which seems promising for the applicability of DCN models to future neuroscientiﬁc investigations regarding the emergence of RF sizes.
Revised: As a further link with biological models, we have demonstrated that the learned scale distributions in DCNs are compatible with experimental observations, which seems promising for the applicability of DCN models to future neuroscientiﬁc investigations regarding the emergence of RF sizes.

Original: In addition, we have presented the capability of DCNs to reduce errors in feature maps caused by masking.
Revised: In addition, we have presented the capability of DCNs to reduce errors in feature maps caused by masking.

Original: Finally, we have empirically shown an interplay between the input contrast to ODE blocks and the time scales of the solutions, which can be capitalized on for computational savings.
Revised: Finally, we have empirically shown an interplay between the input contrast to ODE blocks and the time scales of the solutions, which can be capitalized on for computational savings.

Original: However, one of the biggest limitations of DCNs is that they may become unstable during training.
Revised: However, one of the biggest limitations of DCNs is that they may become unstable during training.

Original: This is a combined problem of neural ODE models and scale ﬁtting, which may lead to exploding ﬁlter sizes with larger learning rates.
Revised: This is a combined problem of neural ODE models and scale ﬁtting, which may lead to exploding ﬁlter sizes with larger learning rates.

Original: Especially for meta-parametrization, it would be advisable to clip the integration time and ﬁlter parameters within a reasonable range.
Revised: Especially for meta-parametrization, it would be advisable to clip the integration time and ﬁlter parameters within a reasonable range.

Original: Nevertheless, we believe there are exciting future research opportunities involving DCNs.
Revised: Nevertheless, we believe there are exciting future research opportunities involving DCNs.

Original: A rigorous mathematical analysis of the nonlinear system of ODEs is beyond the scope of this paper, however, neural ODE formulations provide interesting opportunities for establishing a theoretical understanding of DNNs based on dynamical systems.
Revised: A rigorous mathematical analysis of the nonlinear system of ODEs is beyond the scope of this paper, however, neural ODE formulations provide interesting opportunities for establishing a theoretical understanding of DNNs based on dynamical systems.

Original: The interplay of input contrast and integration time is one such observation which requires further scrutiny.
Revised: The interplay of input contrast and integration time is one such observation which requires further scrutiny.

Original: Similarly, our choice of ﬁlters based on well-behaved Gaussian derivatives allow for further analytical investigations, unlike conventional CNNs.
Revised: Similarly, our choice of ﬁlters based on well-behaved Gaussian derivatives allow for further analytical investigations, unlike conventional CNNs.

Original: Similarly, DCNs offer interesting possibilities for biological modelling.
Revised: Similarly, DCNs offer interesting possibilities for biological modelling.

Original: The in-built smooth evolution of ﬁlters can be used, for example, to incorporate response dynamics such as synaptic depression or short-term potentiation.
Revised: The in-built smooth evolution of ﬁlters can be used, for example, to incorporate response dynamics such as synaptic depression or short-term potentiation.

Original: Likewise, the equations of motion can be modiﬁed to reﬂect axonal delays or generate oscillations.
Revised: Likewise, the equations of motion can be modiﬁed to reﬂect axonal delays or generate oscillations.

Original: In short, we believe by offering a link between dynamical systems, biological models and CNNs, DCNs display an interesting potential to bring together ideas from both ﬁelds.
Revised: In short, we believe by offering a link between dynamical systems, biological models and CNNs, DCNs display an interesting potential to bring together ideas from both ﬁelds.

Original: A A PPENDIX A. G AUSSIAN MULTISCALE LOCAL N- JET
Revised: A A PPENDIX A. G AUSSIAN MULTISCALE LOCAL N- JET

Original: Based on the Schwartz theory of smooth test functions the Gaussian scale-space paradigm states that the derivatives L i 1 ...
Revised: Based on the Schwartz theory of smooth test functions the Gaussian scale-space paradigm states that the derivatives L i 1 ...

Original: i n ( (cid:126)x ; σ ) of a function L 0 ( (cid:126)x ) with respect to the spatial variables x i with i = 1 . . .
Revised: i n ( (cid:126)x ; σ ) of a function L 0 ( (cid:126)x ) with respect to the spatial variables x i with i = 1 . . .

Original: d and at scale σ is given by the convolution
Revised: d and at scale σ is given by the convolution

Original: [Equation]
Revised: [Equation]

Original: (4) where ∂
Revised: (4) where ∂

Original: i 1 ...i n is the n -th order partial derivative and G ( (cid:126)x ; σ ) is the normalized, isotropic Gaussian kernel with standard deviation σ i = σ and mean µ i = 

Revised: i 1 ...i n is the n -th order partial derivative and G ( (cid:126)x ; σ ) is the normalized, isotropic Gaussian kernel with standard deviation σ i = σ and mean µ i =

Original: (Florack et al., 1996).
Revised: (Florack et al., 1996).

Original: Note that L 0 ( (cid:126)x ) does not need to be a smooth function and therefore the Gaussian scale-space paradigm can be applied to obtain local image derivatives in different scales, where (cid:126)x denote the spatial coordinates and σ can be interpreted as the coordinate in the scale dimension.
Revised: Note that L 0 ( (cid:126)x ) does not need to be a smooth function, and therefore the Gaussian scale-space paradigm can be applied to obtain local image derivatives in different scales, where (cid:126)x denote the spatial coordinates and σ can be interpreted as the coordinate in the scale dimension.

Original: We build upon this deﬁnition of local image derivatives to build our structured receptive ﬁelds (SRFs), similar to Jacobsen et al.
Revised: We build upon this deﬁnition of local image derivatives to build our structured receptive ﬁelds (SRFs), similar to Jacobsen et al.

Original: The main idea we leverage is that using a Taylor approximation, one can decompose an input image into a superimposition of its local derivatives.
Revised: The main idea we leverage is that using a Taylor approximation, one can decompose an input image into a superimposition of its local derivatives.

Original: This decomposition can then be performed by local convolution kernels in CNNs, where the relative weight of each derivative order can be learned during training.
Revised: This decomposition can then be performed by local convolution kernels in CNNs, where the relative weight of each derivative order can be learned during training.

Original: In order to show this, we observe that the N -th order Taylor expansion of an image L : R 2 → R around a point ( a, b ) is given by
Revised: In order to show this, we observe that the N -th order Taylor expansion of an image L : R 2 → R around a point ( a, b ) is given by

Original: [Equation]
Revised: [Equation]

Original: where the partial derivatives of L with respect to x and y can be interpreted as L 1 1 ... 1 n ( x, y ; σ 0 ) and L 2 1 ... 2 n ( x, y ; σ 0 ) from equation 4 at some original scale σ 0 .
Revised: where the partial derivatives of L with respect to x and y can be interpreted as L 1 1 ... 1 n ( x, y ; σ 0 ) and L 2 1 ... 2 n ( x, y ; σ 0 ) from equation 4 at some original scale σ 0 .

Original: This means that, via the linearity of convolution, and under the assumption that L ( x, y ; σ ) does not diverge, it is equivalent to either use the N -th order derivatives of the input image, or use the N -th order derivatives of the Gaussian function G ( (cid:126)x ; σ ) , to perform the decomposition at scale σ .
Revised: This means that, via the linearity of convolution, and under the assumption that L ( x, y ; σ ) does not diverge, it is equivalent to either use the N -th order derivatives of the input image, or use the N -th order derivatives of the Gaussian function G ( (cid:126)x ; σ ) , to perform the decomposition at scale σ .

Original: The local N-jets can thus be seen to be parametrized by the coefﬁcients in the expansion in equation 5.
Revised: The local N-jets can thus be seen to be parametrized by the coefﬁcients in the expansion in equation 5.

Original: By deﬁnition, we consider the Taylor expansion coefﬁcients to be covariant derivatives of the image
Revised: By deﬁnition, we consider the Taylor expansion coefﬁcients to be covariant derivatives of the image

Original: L ( x, y ; σ ) , which are independent of the local coordinate system, to reach the ﬁlter approximations given in equation 1, where the coefﬁcients take the form α i,j .
Revised: L ( x, y ; σ ) , which are independent of the local coordinate system, to reach the ﬁlter approximations given in equation 1, where the coefﬁcients take the form α i,j .

Original: In short, the multiscale local N-jet provides a natural decomposition of an image in a local neighborhood in the spatial and scale dimensions.
Revised: In short, the multiscale local N-jet provides a natural decomposition of an image in a local neighborhood in the spatial and scale dimensions.

Original: Under convolution with G ( (cid:126)x ; σ ) , this decomposition provides a framework for deﬁning convolutional ﬁlters in a CNN using N -th order Taylor polynomials.
Revised: Under convolution with G ( (cid:126)x ; σ ) , this decomposition provides a framework for deﬁning convolutional ﬁlters in a CNN using N -th order Taylor polynomials.

Original: The SRF ﬁlters we use are based on this N-jet deﬁnition and allow us to learn the scale, spatial frequency and orientation of the ﬁlters during training, which are fundamental properties of biological receptive ﬁelds (Jones & Palmer, 1987; Lindeberg, 1993).
Revised: The SRF ﬁlters we use are based on this N-jet deﬁnition and allow us to learn the scale, spatial frequency and orientation of the ﬁlters during training, which are fundamental properties of biological receptive ﬁelds (Jones & Palmer, 1987; Lindeberg, 1993).

Original: In addition, the Gaussian scale-space formulation of SRFs (Jacobsen et al., 2016) lead to theoretically interesting properties which strengthen the motivation for our choice of ﬁlters based on the
Revised: In addition, the Gaussian scale-space formulation of SRFs (Jacobsen et al., 2016) lead to theoretically interesting properties which strengthen the motivation for our choice of ﬁlters based on the

Original: Gaussian N-jet.
Revised: Gaussian N-jet.

Original: For example, the semi-group property of Gaussian scale-spaces indicates for a
Revised: For example, the semi-group property of Gaussian scale-spaces indicates for a

Original: Gaussian derivative kernel G i,j ( x, y ; t ) parametrized by the variance t = σ
Revised: Gaussian derivative kernel G i,j ( x, y ; t ) parametrized by the variance t = σ

Original: [Equation]
Revised: [Equation]

Original: where the superscripts ( i, j ) denote the partial derivatives with respect to x and y .
Revised: where the superscripts ( i, j ) denote the partial derivatives with respect to x and y .

Original: This means that a translation in scale dimension can be achieved through convolution with a simple (0-th order)
Revised: This means that a translation in scale dimension can be achieved through convolution with a simple (0-th order)

Original: Gaussian kernel.
Revised: Gaussian kernel.

Original: For DCNs, this means that we have a solid understanding of the scale of the feature maps (in the sense of the Gaussian scale-space) at every layer in the network, as long as we know the learned value of the σ s. In the absence of SRFs with an explicit scale parameter, this information is lost.
Revised: For DCNs, this means that we have a solid understanding of the scale of the feature maps (in the sense of the Gaussian scale-space) at every layer in the network, as long as we know the learned value of the σ s. In the absence of SRFs with an explicit scale parameter, this information is lost.

Original: Similarly, SRF ﬁlters based on Gaussian derivatives are steerable by the coefﬁcients a i,j .
Revised: Similarly, SRF ﬁlters based on Gaussian derivatives are steerable by the coefﬁcients a i,j .

Original: For example, a ﬁlter with orientation θ can be described using second order basis functions as
Revised: For example, a ﬁlter with orientation θ can be described using second order basis functions as

Original: [Equation]
Revised: [Equation]

Original: Finally, the set of Gaussian derivative basis functions G i,j ( x, y ; σ ) are spatially separable
Revised: Finally, the set of Gaussian derivative basis functions G i,j ( x, y ; σ ) are spatially separable

Original: [Equation]
Revised: [Equation]

Original:  which is useful for computational efﬁciency in numerical applications.
Revised: ) which is useful for computational efﬁciency in numerical applications.

Original: A. T RAINING PROCEDURE
Revised: A. T RAINING PROCEDURE

Original: The basic architecture of all our models is given in ﬁgure 2.
Revised: The basic architecture of all our models is given in ﬁgure 2.

Original: Unless otherwise stated, in all models we use group normalization (Wu & He, 2018) with 32 groups in every layer as the normalization function.
Revised: Unless otherwise stated, in all models we use group normalization (Wu & He, 2018) with 32 groups in every layer as the normalization function.

Original: As the nonlinear activation, we use continuously differentiable exponential linear units (Barron, 2017), or CELU for DCN models.
Revised: As the nonlinear activation, we use continuously differentiable exponential linear units (Barron, 2017), or CELU for DCN models.

Original: Based on the original ODE-Net implementation (Chen et al., 2018), for ODE-Net models and ResNet-block models, we use rectiﬁed linear units (ReLU).
Revised: Based on the original ODE-Net implementation (Chen et al., 2018), for ODE-Net models and ResNet-block models, we use rectiﬁed linear units (ReLU).

Original: Likewise, when deﬁning the integration time interval T , we stick to the original implementation, with
Revised: Likewise, when deﬁning the integration time interval T , we stick to the original implementation, with

Original: T = 1 for ODE-Net models, whereas we use T = 2 for DCN models.
Revised: T = 1 for ODE-Net models, whereas we use T = 2 for DCN models.

Original: Otherwise, all the hyperparameters are kept constant between models.
Revised: Otherwise, all the hyperparameters are kept constant between models.

Original: For a brief overview of hyperparameter optimization, see appendix A.3.
Revised: For a brief overview of hyperparameter optimization, see appendix A.3.

Original: All models are trained for 100 epochs on the standard CIFAR-10 training set (Krizhevsky, 2009) using cross-entropy loss.
Revised: All models are trained for 100 epochs on the standard CIFAR-10 training set (Krizhevsky, 2009) using cross-entropy loss.

Original: As data augmentation, we use random translations up to 4 pixels in each spatial dimension and random horizontal ﬂips.
Revised: As data augmentation, we use random translations up to 4 pixels in each spatial dimension and random horizontal ﬂips.

Original: Optimization is performed using SGD with a minibatch size of 128, initial learning rate 10 − 1 , momentum 0.9, and a learning rate decay by a factor of 0.1 at epochs 40 and 70.
Revised: Optimization is performed using SGD with a minibatch size of 128, initial learning rate 10 − 1 , momentum 0.9, and a learning rate decay by a factor of 0.1 at epochs 40 and 70.

Original: For continuous time models based on neural ODEs, we use the adjoint method for backpropagating the losses and ODE solvers with error tolerance set to 10 − 3 .
Revised: For continuous time models based on neural ODEs, we use the adjoint method for backpropagating the losses and ODE solvers with error tolerance set to 10 − 3 .

Original: For convolutional layers with pixel-based k × k ﬁlters, the weights are initialized using the standard
Revised: For convolutional layers with pixel-based k × k ﬁlters, the weights are initialized using the standard

Original: Kaiming uniform initialization (He et al., 2015).
Revised: Kaiming uniform initialization (He et al., 2015).

Original: For layers using SRF ﬁlters, the initial α values are randomly sampled from a normal distribution with mean 0 and standard deviation 0.1, and initial σ values are sampled from a normal distribution with mean 0 and standard deviation 2/3.
Revised: For layers using SRF ﬁlters, the initial α values are randomly sampled from a normal distribution with mean 0 and standard deviation 0.1, and initial σ values are sampled from a normal distribution with mean 0 and standard deviation 2/3.

Original: For the restricted CIFAR-10 experiments (small-data regime), we pick the total number of training images to match our mini-batch size of 128, or otherwise have a minimal number of samples in the ﬁnal batch (which is dropped in each epoch).
Revised: For the restricted CIFAR-10 experiments (small-data regime), we pick the total number of training images to match our mini-batch size of 128, or otherwise have a minimal number of samples in the ﬁnal batch (which is dropped in each epoch).

Original: In order to conﬁrm convergence, for the training set sizes [520 , 1030 , 5120] , we increased the number of training epochs by a factor of [10 , 5 , 2] respectively.
Revised: In order to conﬁrm convergence, for the training set sizes [520 , 1030 , 5120] , we increased the number of training epochs by a factor of [10 , 5 , 2] respectively.

Original: For meta-parametrized models the initial values for the learnable parameters follow normal distributions N ( µ, σN ) : a ∼ N (0 , 2 / 3) , b ∼ N (0 , 0 . 1) , for the DCN σ ( t ) model; a ∼ N (0 , 2 / 3) , b ∼ N (0 , 2 / 3) , c ∼ N (0 , 0 . 1) for the DCN σ ( t 2 ) model; and a s ∼ N (0 , 2 / 3) , b s ∼ N (0 , 0 .
Revised: For meta-parametrized models the initial values for the learnable parameters follow normal distributions N ( µ, σ

Original: For all our models using SRF ﬁlters, except for the DCN σ ij model, we use scale sharing within a convolutional layer such that σ mij =
Revised: For all our models using SRF ﬁlters, except for the DCN σ ij model, we use scale sharing within a convolutional layer such that σ mij =

Original: σ m for all m .
Revised: σ m for all m .

Original: This makes the GPU implementation trivial, as all ﬁlters within a layer are sampled in the interval [ − 2 σ m , 2 σ m ] , and hence the convolutional kernel sizes within a layer are uniform.
Revised: This makes the GPU implementation trivial, as all ﬁlters within a layer are sampled in the interval [ − 2 σ m , 2 σ m ] , and hence the convolutional kernel sizes within a layer are uniform.

Original: However, for the DCN σ ij model, a GPU implementation would be highly inefﬁcient if we truncated the kernels at a ﬁxed factor of σ ij , independently for each input and output channel i, j .
Revised: However, for the DCN σ ij model, a GPU implementation would be highly inefﬁcient if we truncated the kernels at a ﬁxed factor of σ ij , independently for each input and output channel i, j .

Original: Therefore for the DCN σ ij model we ﬁx the kernel size at 7 × 7 , but we still learn the shape and the scale (bandwidth) of the ﬁlters during training.
Revised: Therefore for the DCN σ ij model we ﬁx the kernel size at 7 × 7 , but we still learn the shape and the scale (bandwidth) of the ﬁlters during training.

Original: We use the Dormand–Prince (DOPRI) method (Dormand & Prince, 1980) as the numerical ODE solver.
Revised: We use the Dormand–Prince (DOPRI) method (Dormand & Prince, 1980) as the numerical ODE solver.

Original: The DOPRI method is an explicit, adaptive solver of the Runge-Kutta family, which uses function evaluations to compute fourth- and ﬁfth-order accurate solutions to ODEs, along with an error estimate.
Revised: The DOPRI method is an explicit, adaptive solver of the Runge-Kutta family, which usesfunction evaluations to compute fourth- and ﬁfth-order accurate solutions to ODEs, along with an error estimate.

Original: The size of the adaptive steps taken by the solver can be regulated by specifying an error tolerance on this error estimate.
Revised: The size of the adaptive steps taken by the solver can be regulated by specifying an error tolerance on this error estimate.

Original: As is the case with most modern ODE solvers, the GPU implementation of the DOPRI solver that we use (Chen et al., 2018) considers the input arguments for the integration time interval t ∈ [0 , T ] (or t ∈ [0 , 2] in the case of all DCN models) as soft bounds.
Revised: As is the case with most modern ODE solvers, the GPU implementation of the DOPRI solver that we use (Chen et al., 2018) considers the input arguments for the integration time interval t ∈ [0 , T ] (or t ∈ [0 , 2] in the case of all DCN models) as soft bounds.

Original: This means that the DOPRI algorithm may explore time points outside of this interval, based on its internal error estimation, and may then employ interpolation to return solutions within the speciﬁed bounds.
Revised: This means that the DOPRI algorithm may explore time points outside of this interval, based on its internal error estimation, and may then employ interpolation to return solutions within the speciﬁed bounds.

Original: In the meta-parametrized models, where for example σ is an explicit function of t , this may lead to very large or very small kernel sizes, ordinarily unexpected within the integration time interval.
Revised: In the meta-parametrized models, where for example σ is an explicit function of t , this may lead to very large or very small kernel sizes, ordinarily unexpected within the integration time interval.

Original: In order to avoid numerical instability and memory issues in the meta-parametrized models, we scale down and clip the integration time t when passing it into the parameter deﬁnitions as σ ( τt clip ) and α ( τt clip ) .
Revised: In order to avoid numerical instability and memory issues in the meta-parametrized models, we scale down and clip the integration time t when passing it into the parameter deﬁnitions as σ ( τt clip ) and α ( τt clip ) .

Original: We clip the t values in the interval [ − 0 .
Revised: We clip the t values in the interval [ − 0 .

Original: 5 , 2 . 5] and use τ = 0 .
Revised: 5 , 2 . 5] and use τ = 0 .

Original: A. H YPERPARAMETER TUNING
Revised: A. H YPERPARAMETER TUNING

Original: As mentioned in appendix A.2, we share all the design choices and hyperparameters between all
Revised: As mentioned in appendix A.2, we share all the design choices and hyperparameters between all

Original: DCN and baseline ODE-Net models, except for the nonlinear activation function and integration time interval T .
Revised: DCN and baseline ODE-Net models, except for the nonlinear activation function and integration time interval T .

Original: This difference in design choices arises since for the ODE-Net baseline we stay faithful to the original ODE-Net implementation, where ReLU is the nonlinear activation function and T = 1 , whereas for DCN models we use CELU as the activation function, due to its generalized compatibility with the adjoint method, and T = 2 .
Revised: This difference in design choices arises since for the ODE-Net baseline we stay faithful to the original ODE-Net implementation, where ReLU is the nonlinear activation function and T = 1 , whereas for DCN models we use CELU as the activation function, due to its generalized compatibility with the adjoint method, and T = 2 .

Original: In order to verify that our design choices do not provide an unfair advantage over the ODE-Net baseline, we run some control experiments, where we vary the activation function and T in the ODE-Net baseline.
Revised: In order to verify that our design choices do not provide an unfair advantage over the ODE-Net baseline, we run some control experiments, where we vary the activation function and T in the ODE-Net baseline.

Original: We ﬁnd that the change of activation functions or integration interval T do not provide a signiﬁcant increase to the CIFAR-10 classiﬁcation performance in the ODE-Net baseline (Table 5).
Revised: We ﬁnd that the change of activation functions or integration interval T do not provide a signiﬁcant increase to the CIFAR-10 classiﬁcation performance in the ODE-Net baseline (Table 5).

Original: [Table]
Revised: [Table]

Original: A. CIFAR-10 IMAGE RECONSTRUCTION
Revised: A. CIFAR-10 IMAGE RECONSTRUCTION

Original: For the reconstruction task, we use the network architectures with 3 DCN-ODE blocks (Fig. 2), or for the baseline networks ODE-Net or ResNet-blocks, trained on the CIFAR-10 classiﬁcation task as an encoder.
Revised: For the reconstruction task, we use the network architectures with 3 DCN-ODE blocks (Fig. 2), or for the baseline networks ODE-Net or ResNet-blocks, trained on the CIFAR-10 classiﬁcation task as an encoder.

Original: We use the feature maps at the end of ODE Block 3 as the input to a small decoder network.
Revised: We use the feature maps at the end of ODE Block 3 as the input to a small decoder network.

Original: The decoder DCN network is made up of an ODE block with 128 input channels, followed by bilinear upscaling, a 1 × 1 convolution to reduce the number of channels to 64, another ODE block, followed by bilinear upscaling, normalization, non-linear activation and a 1 × 1 convolution to generate the output in RGB space.
Revised: The decoder DCN network is made up of an ODE block with 128 input channels, followed by bilinear upscaling, a 1 × 1 convolution to reduce the number of channels to 64, another ODE block, followed by bilinear upscaling, normalization, non-linear activation and a 1 × 1 convolution to generate the output in RGB space.

Original: We implement reconstruction as a regression task and use the mean squred error (MSE) as the loss function.
Revised: We implement reconstruction as a regression task and use the mean squred error (MSE) as the loss function.

Original: Otherwise, the training parameters are the same as the classiﬁcation experiments: We usethe SGD optimizer with a mini-batch size of 128, learning rate 10 − 1 and momentum 0.9, together with the adjoint method and an error tolerance of 10 − 3 . The reconstruction of some example images (randomly selected) from the CIFAR-10 validation set by the DCN and baseline networks are shown in ﬁgure 5.
Revised: Otherwise, the training parameters are the same as the classiﬁcation experiments: We use

Original: [Figure]
Revised: [Figure]

Original: -ODE

Original: ODE-Net
Revised: ODE-Net

Original: ResNetBlocks 
Revised: ResNetBlocks the SGD optimizer with a mini-batch size of 128, learning rate 10 − 1 and momentum 0.9, together with the adjoint method and an error tolerance of 10 − 3 . The reconstruction of some example images (randomly selected) from the CIFAR-10 validation set by the DCN and baseline networks are shown in ﬁgure 5.

Original: A. P ATTERN COMPLETION IN FEATURE MAPS
Revised: A. P ATTERN COMPLETION IN FEATURE MAPS

Original: In ﬁgure 6, we show the feature map evolution within the ﬁrst ODE block (or ResNet block) of different models with and without masking of some example input images.
Revised: In ﬁgure 6, we show the feature map evolution within the ﬁrst ODE block (or ResNet block) of different models with and without masking of some example input images.

Original: Size of the mask depicted here is 6 × 6 pixels and the example images were chosen so as to have the mask located close to the middle of the object.
Revised: Size of the mask depicted here is 6 × 6 pixels and the example images were chosen so as to have the mask located close to the middle of the object.

Original: We picked some channels with visible mask-related artifacts in the input feature maps to the ﬁrst ODE (ResNet) block.
Revised: We picked some channels with visible mask-related artifacts in the input feature maps to the ﬁrst ODE (ResNet) block.

Original: Since there is no feature map trajectory in the ResNet-blocks model, but only one input and one output feature map, the difference between the feature maps of the intact and masked image is given as a scatter plot of two data points connected by a red line.
Revised: Since there is no feature map trajectory in the ResNet-blocks model, but only one input and one output feature map, the difference between the feature maps of the intact and masked image is given as a scatter plot of two data points connected by a red line.

Original: Figure 7 depicts the average difference of intact and masked feature maps D ( t ) averaged overimages and the standard deviation for the DCN and baseline networks.
Revised: Figure 7 depicts the average difference of intact and masked feature maps D ( t ) averaged overimages and the standard deviation for the DCN and baseline networks.

Original: [Figure]
Revised: [Figure]

Original: ) h
Revised: ) h

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]