Original: C A LFADS: LATENT FACTOR ANALYSIS OF DYNAMICAL SYSTEMS IN CALCIUM IMAGING DATA
Revised: C A LFADS: LATENT FACTOR ANALYSIS OF DYNAMICAL SYSTEMS IN CALCIUM IMAGING DATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Dynamic latent variable modelling has provided a powerful tool for understanding how populations of neurons compute.
Revised: Dynamic latent variable modelling has provided a powerful tool for understanding how populations of neurons compute.

Original: For spiking data, such latent variable modelling can treat the data as set of point-processes, due to the fact that spiking dynamics occur on a much faster timescale than the computational dynamics being inferred.
Revised: For spiking data, such latent variable modelling can treat the data as set of point-processes, due to the fact that spiking dynamics occur on a much faster timescale than the computational dynamics being inferred.

Original: In contrast, for other experimental techniques the slow dynamics governing the observed data are similar in timescale to the computational dynamics that researchers want to infer.
Revised: In contrast, for other experimental techniques the slow dynamics governing the observed data are similar in timescale to the computational dynamics that researchers want to infer.

Original: An example of this is in calcium imaging data, where calcium dynamics can have time timescales on the order of hundreds of milliseconds.
Revised: An example of this is in calcium imaging data, where calcium dynamics can have time timescales on the order of hundreds of milliseconds.

Original: As such, the successful application of dynamic latent variable modelling to modalities like calcium imaging data will rest on the ability to disentangle the higher- and shallower-level dynamical systems’ contributions to the data.
Revised: As such, the successful application of dynamic latent variable modelling to modalities like calcium imaging data will rest on the ability to disentangle the higher- and shallower-level dynamical systems’ contributions to the data.

Original: To-date, no techniques have been developed to directly achieve this.
Revised: To-date, no techniques have been developed to directly achieve this.

Original: Here we solve this problem by extending recent advances using sequential variational autoencoders for dynamic latent variable modelling of neural data.
Revised: Here we solve this problem by extending recent advances using sequential variational autoencoders for dynamic latent variable modelling of neural data.

Original: We solve the problem of disentangling higher
Revised: We solve the problem of disentangling higher

Original: - and shallower-level dynamics by incorporating a ladder architecture that can infer a hierarchy of dynamical systems
Revised: - and shallower-level dynamics by incorporating a ladder architecture that can infer a hierarchy of dynamical systems

Original: Using some built
Revised: Using some built

Original: -in inductive biases for calcium dynamics, we show that we can disentangle calcium ﬂux from the underlying dynamics of neural computation.
Revised: -in inductive biases for calcium dynamics, we show that we can disentangle calcium ﬂux from the underlying dynamics of neural computation.

Original: First
Revised: First

Original: , we demonstrate with synthetic calcium data that we can correctly disentangle an un derlying Lorenz attractor from calcium dynamics
Revised: , we demonstrate with synthetic calcium data that we can correctly disentangle an un derlying Lorenz attractor from calcium dynamics

Original: Next
Revised: Next

Original: , we show that we can infer appropriate rotational dynamics in spiking data from macaque motor cortex after it has been converted into calcium ﬂuorescence data via a calcium dynam ics model
Revised: , we show that we can infer appropriate rotational dynamics in spiking data from macaque motor cortex after it has been converted into calcium ﬂuorescence data via a calcium dynam ics model

Original: Finally
Revised: Finally

Original: , we show that our method applied to real calcium imaging data from primary visual cortex in mice allows us to infer latent factors that carry salient sensory information about unexpected stimuli.
Revised: , we show that our method applied to real calcium imaging data from primary visual cortex in mice allows us to infer latent factors that carry salient sensory information about unexpected stimuli.

Original: These results demonstrate that variational ladder autoencoders are a promising approach for inferring hier
Revised: These results demonstrate that variational ladder autoencoders are a promising approach for inferring hier

Original: - archical dynamics in experimental settings where the measured variable has its own slow dynamics, such as calcium imaging data.
Revised: - archical dynamics in experimental settings where the measured variable has its own slow dynamics, such as calcium imaging data.

Original: Our new, open source tool, thereby provides the neuroscience community with the ability to apply dynamic latent variable modelling to a wider array of data modalities.
Revised: Our new, open source tool, thereby provides the neuroscience community with the ability to apply dynamic latent variable modelling to a wider array of data modalities.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Dynamic latent variable modelling has been a hugely successful approach to understanding the function of neural circuits.
Revised: Dynamic latent variable modelling has been a hugely successful approach to understanding the function of neural circuits.

Original: For example, it has been used to uncover previously unknown mechanisms for computation in the motor cortex (Churchland et al., 2012; Pandarinath et al., 2018), somatosensory cortex (Wei et al., 2019b), and hippocampus (Chaudhuri et al., 2019).
Revised: For example, it has been used to uncover previously unknown mechanisms for computation in the motor cortex (Churchland et al., 2012; Pandarinath et al., 2018), somatosensory cortex (Wei et al., 2019b), and hippocampus (Chaudhuri et al., 2019).

Original: However, the success of this approach is largely limited to datasets where the observed variables have dynamics whose timescales are much faster than the dynamics of the underlying computations.
Revised: However, the success of this approach is largely limited to datasets where the observed variables have dynamics whose timescales are much faster than the dynamics of the underlying computations.

Original: This is the case, for example, with spiking data, where the dynamics governing the generation of individual spikes are much faster than the dynamics of computation across the circuit.
Revised: This is the case, for example, with spiking data, where the dynamics governing the generation of individual spikes are much faster than the dynamics of computation across the circuit.

Original: This makes it possible to characterise the observed data, e.g. the spiking data, as a set of point-processes that can be used directly for inferring latent variables.
Revised: This makes it possible to characterise the observed data, e.g. the spiking data, as a set of point-processes that can be used directly for inferring latent variables.

Original: However, many datasets in the life sciences are generated by a hierarchy of dynamical systems, wherein the shallower-level dynamical systems that directly generate the observed data have temporal dynamics whose timescales overlap with that of the deeper-level dynamical system to be inferred.
Revised: However, many datasets in the life sciences are generated by a hierarchy of dynamical systems, wherein the shallower-level dynamical systems that directly generate the observed data have temporal dynamics whose timescales overlap with that of the deeper-level dynamical system to be inferred.

Original: A clear example of this is in-vivo calcium imaging data, which is widely used in neuroscience.
Revised: A clear example of this is in-vivo calcium imaging data, which is widely used in neuroscience.

Original: Thanks to advances in imaging technology and genetically encoded calcium indicators, calcium imaging enables monitoring of the activity of large populations of genetically targeted neurons in awake behaving animals (Yang & Yuste, 2017; Lin & Schnitzer, 2016).
Revised: Thanks to advances in imaging technology and genetically encoded calcium indicators, calcium imaging enables monitoring of the activity of large populations of genetically targeted neurons in awake behaving animals (Yang & Yuste, 2017; Lin & Schnitzer, 2016).

Original: However, calcium imaging introduces an additional layer of a relatively slow dynamical system between the computations occurring in the brain and the measurements that neuroscientists make.
Revised: However, calcium imaging introduces an additional layer of a relatively slow dynamical system between the computations occurring in the brain and the measurements that neuroscientists make.

Original: This problem is outlined in
Revised: This problem is outlined in

Original: Figure 1A, in which calcium ﬂuorescence observations, x , depend on the state of calcium ﬂux, z 1 , which is governed by a shallower-level dynamical system with temporal dynamics on the order of hundreds of milliseconds.
Revised: Figure 1A, in which calcium ﬂuorescence observations, x , depend on the state of calcium ﬂux, z 1 , which is governed by a shallower-level dynamical system with temporal dynamics on the order of hundreds of milliseconds.

Original: These dynamics are driven, in part, by perturbations due to spikes, u 1 , which are themselves governed by a computational state, z 2 , with a similar timescale in its dynamics to the calcium ﬂux (and which itself can be perturbed independently by unknown inputs, u 2 , that may also have slow dynamics).
Revised: These dynamics are driven, in part, by perturbations due to spikes, u 1 , which are themselves governed by a computational state, z 2 , with a similar timescale in its dynamics to the calcium ﬂux (and which itself can be perturbed independently by unknown inputs, u 2 , that may also have slow dynamics).

Original: Due to the overlap in timescales, it is impossible to identify immediately which components of the calcium ﬂuorescence data are driven by the dynamics of calcium ﬂux, and which are driven by the deeper-level latent dynamics of neural computation.
Revised: Due to the overlap in timescales, it is impossible to identify immediately which components of the calcium ﬂuorescence data are driven by the dynamics of calcium ﬂux, and which are driven by the deeper-level latent dynamics of neural computation.

Original: Ideally, neuroscientists would have a method for inferring both the shallower-level calcium dynamics and the deeper-level computational dynamics in order to uncover the hierarchical dynamical systems that generated their data.
Revised: Ideally, neuroscientists would have a method for inferring both the shallower-level calcium dynamics and the deeper-level computational dynamics in order to uncover the hierarchical dynamical systems that generated their data.

Original: Such a tool would be of signiﬁcant beneﬁtto the systems neuroscience community.
Revised: Such a tool would be signiﬁcantly beneﬁt the systems neuroscience community.

Original: Currently, these problems are treated separately.
Revised: Currently, these problems are treated separately.

Original: For situations where the observed data can be treated as a point process, we have good techniques for inferring the deeper-level dynamics.
Revised: For situations where the observed data can be treated as a point process, we have good techniques for inferring the deeper-level dynamics.

Original: For example, recent applications of sequential variational autoencoders have seen great success in inferring underlying computations from extracellular spiking data (Pandarinath et al., 2018).
Revised: For example, recent applications of sequential variational autoencoders have seen great success in inferring underlying computations from extracellular spiking data (Pandarinath et al., 2018).

Original: This technique, named Latent Factor Analysis of Dynamical Systems (LFADS), has improved neuroscientists’ ability to infer underlying neural computations from spiking data, e.g. it has been used to identify latent rotational reaching dynamics and to decode reaching behaviour of macaques and humans with higher ﬁdelity than other techniques.
Revised: This technique, named Latent Factor Analysis of Dynamical Systems (LFADS), has improved neuroscientists’ ability to infer underlying neural computations from spiking data, e.g. it has been used to identify latent rotational reaching dynamics and to decode reaching behaviour of macaques and humans with higher ﬁdelity than other techniques.

Original: Although LFADS was a signiﬁcant advance in our ability to analyze neural data in the form of spike trains, it does not address the problem highlighted above for calcium imaging, wherein the calcium dynamics introduce an additional shallower-level dynamical system whose timescale overlaps with the timescale of neural computation.
Revised: Although LFADS has signiﬁcantly advanced our ability to analyze neural data in the form of spike trains, it does not address the problem highlighted above for calcium imaging, wherein the calcium dynamics introduce an additional shallower-level dynamical system whose timescale overlaps with the timescale of neural computation.

Original: Theoretically, this problem could be solved independently by ﬁrst inferring spikes from calcium data, whether by deconvolution (e.g., OASIS) (Friedrich et al., 2017), variational inference (e.g., DeepSpike)
Revised: Theoretically, this problem could be solved independently by ﬁrst inferring spikes from calcium data, whether by deconvolution (e.g., OASIS) (Friedrich et al., 2017), variational inference (e.g., DeepSpike)

Original: (Speiser et al., 2017), dynamic programming (Deneux et al., 2016), or a number of other methods (Berens et al., 2018;
Revised: (Speiser et al., 2017), dynamic programming (Deneux et al., 2016), or any other method (Berens et al., 2018;

Original: Pachitariu et al., 2018; Evans et al., 2019; Wei et al., 2020), and then applying LFADS.
Revised: Pachitariu et al., 2018; Evans et al., 2019; Wei et al., 2020), and then applying LFADS.

Original: However, this approach treats each calcium trace as a completely independent variable when conducting the inference on calcium dynamics, and therefore.
Revised: However, this approach treats each calcium trace as a completely independent variable when inferring calcium dynamics.

Original: This ignores correlations in population activity that could be used to separate calcium dynamics (which is independent of population activity) and computational dynamics (which is not independent of population activity).
Revised: This ignores correlations in population activity that inform the separation of calcium dynamics (which are independent of population activity) from computational dynamics (which are not independent of population activity).

Original: If this separation is sub-optimal, then inference of the deeperlevel system will be impaired.
Revised: If this separation is sub-optimal, then inference of the deeper-level system will be impaired.

Original: Here, we address this problem by extending LFADS with a variational ladder autoencoder architecture (Zhao et al., 2017) that folds the calcium dynamics inference into the larger inference problem (Fig. 1B).
Revised: Here, we address this problem by extending LFADS with a variational ladder autoencoder architecture (Zhao et al., 2017) that folds the calcium dynamics inference into the larger inference problem (Fig. 1B).

Original: Our system, CaLFADS, incorporates inductive biases for calcium dynamics and, thanks to the ladder architecture, is able to infer the deeper-level dynamical system better than an approach that treats inference of calcium dynamics and deeper-level dynamics as separate problems.
Revised: Our system, CaLFADS, incorporates inductive biases for calcium dynamics and, thanks to the ladder architecture, is able to infer the deeper-level dynamical system better than an approach that treats inference of calcium dynamics and deeper-level dynamics as separate problems.

Original: For further discussion on how CaLFADS relates to previous work, see Appendix B.
Revised: For further discussion on how CaLFADS relates to previous work, see Appendix B.

Original: First, we show using synthetic data that we are able to reconstruct ground-truth latent dynamics from synthetic calcium traces.
Revised: First, we show using synthetic data that we are able to reconstruct ground-truth latent dynamics from synthetic calcium traces.

Original: Next, we apply CaLFADS to spiking data from macaque motor cortex that has been converted into calcium ﬂuorescence traces using a calcium dynamics model.
Revised: Next, we apply CaLFADS to spiking data from macaque motor cortex that has been converted into calcium ﬂuorescence traces using a calcium dynamics model.

Original: We show that we are able to recover rotational dynamics from this “calciﬁed” data just as LFADS identiﬁes rotational dynamics from spiking data.
Revised: We show that we are able to recover rotational dynamics from this “calciﬁed” data just as LFADS identiﬁes rotational dynamics from spiking data.

Original: Finally, we show using real 2-photon calcium imaging data from mouse primary visual cortex that CaLFADS can identify deeper-level latent factors that carry information about unexpected visual stimuli.
Revised: Finally, we show using real 2-photon calcium imaging data from mouse primary visual cortex that CaLFADS can identify deeper-level latent factors that carry information about unexpected visual stimuli.

Original: Altogether, our work shows the beneﬁts of incorporating the calcium dynamics inference procedure into the larger inference problem.
Revised: Altogether, our work shows the beneﬁts of incorporating the calcium dynamics inference procedure into the larger inference problem.

Original: It also provides neuroscientists with a new, open-source tool for analyzing calcium imaging data in order to identify deeper-level dynamics.
Revised: It also provides neuroscientists with a new, open-source tool for analyzing calcium imaging data in order to identify deeper-level dynamics.

Original: Given the importance of calcium imaging to modern systems neuroscience, we believe that CaLFADS will be a very useful analysis tool for theneuroscience community.
Revised: Given the importance of calcium imaging to modern systems neuroscience, we believe that CaLFADS will be a very useful analysis tool for the community.

Original: Moreover, we believe that our approach may be applicable to other life sciences do- mains where  theobserved variables have relatively slow dynamics, and thus, could serve as a basis for the construction of future analysis tools.
Revised: Furthermore, our CaLFADS could be adapted to other neuroscience data modalities, such as fMRI data which, like calcium imaging, comprises both fast deeper-level latent brain dynamics and slower shallower-level blood oxygen measurement dynamics (Ollinger et al., 2001). Beyond this, we believe that our ap- proach may be applicable to other life sciences domains, for example insurance claim modelling, where better identifying the relatively slow dynamics underlying the observed variables, such as claim submission distributions over time, could improve forecasting accuracy (Schmidt, 2014).

Original:  M ODEL
Revised:  M ODEL

Original: CaLFADS extends the variational ladder autoencoder (VLAE) approach (Zhao et al., 2017) with
Revised: CaLFADS extends the variational ladder autoencoder (VLAE) approach (Zhao et al., 2017) with

Original: RNNs (Fig 1B), which enables inference of hierarchical latent dynamical systems 1 .
Revised: RNNs (Fig 1B), which enables inference of hierarchical latent dynamical systems 1 .

Original: The full directed acyclic graph for CaLFADS can be found in Figure S1.
Revised: The full directed acyclic graph for CaLFADS can be found in Figure S1.

Original: The VLAE approach to inferring hierarchies removes the sequential dependence of each latent variable, and instead loads parts of a ﬂat latent code with more abstract features using more expressive neural networks (see appendix C for more background on VLAEs).
Revised: The VLAE approach to inferring hierarchies removes the sequential dependence of each latent variable, and instead loads parts of a ﬂat latent code with more abstract features using more expressive neural networks (see appendix C for more background on VLAEs).

Original: For CaLFADS, this means we separate the shallower-level inference and the deeper-level inference into parallel pathways for inference and generation (1B, orange arrow), and subsequently, recombines these levels in the generative network (1B, pink arrow).
Revised: For CaLFADS, this means we separate the shallower-level inference and the deeper-level inference into parallel pathways for inference and generation (1B, orange arrow), and subsequently, recombines these levels in the generative network (1B, pink arrow).

Original: This approach has two major advantages: 1) It solves the problem of latent variable collapse in which all latent features are loaded onto the lowest-order variables in the hierarchy.
Revised: This approach has two major advantages: 1) It solves the problem of latent variable collapse in which all latent features are loaded onto the lowest-order variables in the hierarchy.

Original: It retains a very simple formulation for the variational/evidence lower bound (ELBO) that is easily reparameterisable (Sønderby et al., 2016; Zhao et al., 2017).
Revised: It retains a very simple formulation for the variational/evidence lower bound (ELBO) that is easily reparameterisable (Sønderby et al., 2016; Zhao et al., 2017).

Original: CaLFADS infers the dynamics and unknown events (e.g. spikes) that determine the calcium ﬂuorescence signal (Fig 1B, red network nodes), and the underlying computational dynamical system that controls these events (Fig 1B, blue network nodes), through successive encoding and decoding layers in which latent variable distributions are modelled in separated ‘rungs’ of the ladder architecture.
Revised: CaLFADS infers the dynamics and unknown events (e.g. spikes) that determine the calcium ﬂuorescence signal (Fig 1B, red network nodes), and the underlying computational dynamical system that controls these events (Fig 1B, blue network nodes), through successive encoding and decoding layers in which latent variable distributions are modelled in separated ‘rungs’ of the ladder architecture.

Original: This VLAE architecture enables CaLFADS to disentangle the shallower-level calcium dynamics and the deeper-level computational dynamics.
Revised: This VLAE architecture enables CaLFADS to disentangle the shallower-level calcium dynamics and the deeper-level computational dynamics.

Original: Algorithm 1 provides details on the forward pass of CaLFADS.
Revised: Algorithm 1 provides details on the forward pass of CaLFADS.

Original: Additionally, we provide a complete description of the model in Appendix D. Brieﬂy, layers of Gated Recurrent Units (GRUs) create a deterministic hierarchical embedding of the data in the inference network (Fig 1B, above the dashed line).
Revised: Additionally, we provide a complete description of the model in Appendix D. Brieﬂy, layers of Gated Recurrent Units (GRUs) create a deterministic hierarchical embedding of the data in the inference network (Fig 1B, above the dashed line).

Original: These embeddings are then concatenated with time-shifted, layer-speciﬁc deterministic projections of the latent state.
Revised: These embeddings are then concatenated with time-shifted, layer-speciﬁc deterministic projections of the latent state.

Original: This concatenated set of activities is then linearly transformed into the parameters of a factorised Gaussian posterior distribution for the latent variables in the generative network (Fig 1B, below the dashed line).
Revised: This concatenated set of activities is then linearly transformed into the parameters of a factorised Gaussian posterior distribution for the latent variables in the generative network (Fig 1B, below the dashed line).

Original: This process provides the inference network with information about the history of the latent states via feedback connections (Fig 1B, purple feedback arrows).
Revised: This process provides the inference network with information about the history of the latent states via feedback connections (Fig 1B, purple feedback arrows).

Original: As in any variational autoencoder (VAE) system, latent states are sampled from the posterior distributions determined by the inference network.
Revised: As in any variational autoencoder (VAE) system, latent states are sampled from the posterior distributions determined by the inference network.

Original: For the inference of the computational dynamical system, these latent states represent the initial conditions ( g 2 ,t 0 ) and unknown inputs ( u 2 ,t ) to a learned dynamical system modelled discretely by a GRU in the generative network ( g 2 ,t ).
Revised: For the inference of the computational dynamical system, these latent states represent the initial conditions ( g 2 ,t 0 ) and unknown inputs ( u 2 ,t ) to a learned dynamical system modelled discretely by a GRU in the generative network ( g 2 ,t ).

Original: The hidden state of this GRU is then linearly transformed into a set of potentially low-dimensional time-varying latent factors ( ft ).
Revised: The hidden state of this GRU is then linearly transformed into a set of potentially low-dimensional time-varying latent factors ( ft ).

Original: Concurrently, the latent states for the calcium dynamics ( z t ) are concatenated with the dynamic factors from the layer above ( f t ), and undergo a linear transformation followed by a rectiﬁed-exponential non-linearity f ( x ) = ReLU (exp( x ) − 1) to provide an approximation to spike counts ( u 1 ,t ).
Revised: Concurrently, the latent states for the calcium dynamics ( z t ) are concatenated with the dynamic factors from the layer above ( f t ), and undergo a linear transformation followed by a rectiﬁed-exponential non-linearity f ( x ) = ReLU (exp( x ) − 1) to provide an approximation to spike counts ( u 1 ,t ).

Original: This non-linearity was chosen as a continuous relaxation to discrete calcium transient amplitudes inspired by the static distribution of amplitudes extracted via OASIS (Wei et al., 2019a).
Revised: This non-linearity was chosen as a continuous relaxation to discrete calcium transient amplitudes inspired by the static distribution of amplitudes extracted via OASIS (Wei et al., 2019a).

Original: In our case, we model approximate spike counts using a log-normal distribution with truncation at 0.
Revised: In our case, we model approximate spike counts using a log-normal distribution with truncation at 0.

Original: The approximate spike counts are then used as inputs to an order 1 autoregressive (AR(1)) process model of calcium dynamics ( g 1 ,t ).
Revised: The approximate spike counts are then used as inputs to an order 1 autoregressive (AR(1)) process model of calcium dynamics ( g 1 ,t ).

Original: Since we found that the loss function was extremely sensitive to changes in the AR(1) model parameters, these parameters were ﬁxed using values empirically derived from the OASIS-deconvolution algorithm (Friedrich et al., 2017).
Revised: Since we found that the loss function was extremely sensitive to changes in the AR(1) model parameters, these parameters were ﬁxed using values empirically derived from the OASIS-deconvolution algorithm (Friedrich et al., 2017).

Original: While it is possible to transform calcium dynamics into ﬂuorescence using nonlinear models of ﬂuorescence indicator kinetics (Speiser et al., 2017; Deneux et al., 2016), we found that using the AR(1) process to model the transformation of spike counts to ﬂuorescence was sufﬁcient for accurate reconstruction of the latent space, even in synthetic data with nonlinear calcium transient generation.
Revised: While it is possible to transform calcium dynamics into ﬂuorescence using nonlinear models of ﬂuorescence indicator kinetics (Speiser et al., 2017; Deneux et al., 2016), we found that using the AR(1) process to model the transformation of spike counts to ﬂuorescence was sufﬁcient for accurate reconstruction of the latent space, even in synthetic data with nonlinear calcium transient generation.

Original: Model hyperparameters for different datasets are shown in table S1.
Revised: Model hyperparameters for different datasets are shown in table S1.

Original: 2. L OSS FUNCTION AND T RAINING
Revised: 2. L OSS FUNCTION AND T RAINING

Original: One of the advantages of using VLAEs is that because the stochastic nodes operate in parallel, the
Revised: One of the advantages of using VLAEs is that because the stochastic nodes operate in parallel, the

Original: ELBO formulation is the same as that for standard VAEs despite the hierarchical latent space (Zhao et al., 2017).
Revised: ELBO formulation is the same as that for standard VAEs despite the hierarchical latent space (Zhao et al., 2017).

Original: As such, the CaLFADS cost function remains very similar to that of LFADS.
Revised: As such, the CaLFADS cost function remains very similar to that of LFADS.

Original: It is described in Algorithm 2.
Revised: It is described in Algorithm 2.

Original: One key addition to the LFADS loss function is that because the data is ﬂuorescence data, which includes normal emissions noise, and the inferred shallower-level factors are calcium concentrations, the likelihood function of the data given the estimated calcium concentration, P ( x t | ˆ x t ) , is modelled as a Gaussian distribution
Revised: One key addition to the LFADS loss function is that because the data is ﬂuorescence data, which includes normal emissions noise, and the inferred shallower-level factors are calcium concentrations, the likelihood function of the data given the estimated calcium concentration, P ( x t | ˆ x t ) , is modelled as a Gaussian distribution

Original: x t ∼ N (ˆ x t , σ 2 x ) , where σ 2 x is determined by OASIS.
Revised: x t ∼ N (ˆ x t , σ 2 x ) , where σ 2 x is determined by OASIS.

Original: Another key addition is that although spike counts u 1 ,t are not modelled discretely, their probability distribution given the inferred spike rates, P ( u 1 ,t | λ t ) , is modelled as an approximate
Revised: Another key addition is that although spike counts u 1 ,t are not modelled discretely, their probability distribution given the inferred spike rates, P ( u 1 ,t | λ t ) , is modelled as an approximate

Original: Poisson process such that log( P ( u 1 ,t | λ t ))
Revised: Poisson process such that log( P ( u 1 ,t | λ t ))

Original: ≈ u 1 ,t log( λ t ) − ( λ t + u 1 ,t ) .
Revised: ≈ u 1 ,t log( λ t ) − ( λ t + u 1 ,t ) .

Original: The approximation results from the use of an L1 norm on the spikes, instead of the log-gamma term that would normally be used in a Poisson process, which was necessary because most of the data sits within a range that is poorly handled by the log-gamma function.
Revised: The approximation results from the use of an L1 norm on the spikes, instead of the log-gamma term that would normally be used in a Poisson process, which was necessary because most of the data sits within a range that is poorly handled by the log-gamma function.

Original: The use of this approximate Poisson process provides an additional regularization term in the cost function, which acts as an inductive bias to ensure that the inferred spike trains in the shallower-level dynamical system are constrained to follow the dynamics of the deeper-level dynamical system’s spike rates λ t .
Revised: The use of this approximate Poisson process provides an additional regularization term in the cost function, which acts as an inductive bias to ensure that the inferred spike trains in the shallower-level dynamical system are constrained to follow the dynamics of the deeper-level dynamical system’s spike rates λ t .

Original: Per the VLAE framework, the shallower-level and deeper-level latent variables are treated as independent, with hierarchy imposed in the deterministic paths, i.e., Q ( g 1 ,t , g 2 ,t =0 , u 2 ,t | x ) =
Revised: Per the VLAE framework, the shallower-level and deeper-level latent variables are treated as independent, with hierarchy imposed in the deterministic paths, i.e., Q ( g 1 ,t , g 2 ,t =0 , u 2 ,t | x ) =

Original: Q ( g 1 ,t | x ) Q ( g 2 ,t =

Revised: Q ( g 1 ,t | x ) Q ( g 2 ,t =

Original: | x ) Q ( u 2 ,t | x )
Revised: | x ) Q ( u 2 ,t | x )

Original: Parameters of CaLFADS were optimised with ADAM, with an initial learning rate of 0.01.
Revised: Parameters of CaLFADS were optimised with ADAM, with an initial learning rate of 0.01.

Original: Learning rates were decreased by a factor of 0.95 when plateaus in training error were detected.
Revised: Learning rates were decreased by a factor of 0.95 when plateaus in training error were detected.

Original: As in LFADS training, KL and L2 terms in the cost function were ‘warmed up’, i.e., had a weight w l ∈ (1 , 2) between
Revised: As in LFADS training, KL and L2 terms in the cost function were ‘warmed up’, i.e., had a weight w l ∈ (1 , 2) between

Original: 0 and 1 applied that gradually increased (Bowman et al., 2016;
Revised: 0 and 1 applied that gradually increased (Bowman et al., 2016;

Original: Sønderby et al., 2016).
Revised: Sønderby et al., 2016).

Original: Warm-up for the deeper parameters ( l = 2 , blue modules in Figure 1) was delayed until warm-up for shallower parameters ( l = 1 , red modules in Figure 1) was completed.
Revised: Warm-up for the deeper parameters ( l = 2 , blue modules in Figure 1) was delayed until warm-up for shallower parameters ( l = 1 , red modules in Figure 1) was completed.

Original: We found in preliminary tests that this delayed warm-up sequence was necessary for good hierarchical inference compared to simultaneous warm-up.
Revised: We found in preliminary tests that this delayed warm-up sequence was necessary for good hierarchical inference compared to simultaneous warm-up.

Original: As with aggressive inference network training methods (He et al., 2019), this prevents issues with training of the deeper-level dynamical system parameters converging more slowly than the shallower-level dynamical system parameters in the initial stages of training.
Revised: As with aggressive inference network training methods (He et al., 2019), this prevents issues with training of the deeper-level dynamical system parameters converging more slowly than the shallower-level dynamical system parameters in the initial stages of training.

Original:  R ESULTS 3. L ORENZ A TTRACTOR S YNTHETIC DATA 3.1. D ATA D ESCRIPTION
Revised:  R ESULTS 3. L ORENZ A TTRACTOR S YNTHETIC DATA 3.1. D ATA D ESCRIPTION

Original: As an initial test of CaLFADS, we examined its ability to infer the hierarchical dynamical systems of synthetic calcium ﬂuorescence data generated from a simple model of spiking neurons.
Revised: As an initial test of CaLFADS, we examined its ability to infer the hierarchical dynamical systems of synthetic calcium ﬂuorescence data generated from a simple model of spiking neurons.

Original: We examined synthetic data generated from a network with a Lorenz attractor embedded in its dynamics.
Revised: We examined synthetic data generated from a network with a Lorenz attractor embedded in its dynamics.

Original: This tests our ability to recover a ground-truth deeper-level dynamical system from the data, and has been used as a benchmark by many others (Zhao & Park, 2017; Pandarinath et al., 2018;
Revised: This tests our ability to recover a ground-truth deeper-level dynamical system from the data, and has been used as a benchmark by many others (Zhao & Park, 2017; Pandarinath et al., 2018;

Original: Clark et al., 2019; She & Wu, 2019;
Revised: Clark et al., 2019; She & Wu, 2019;

Original: Koppe et al., 2019).
Revised: Koppe et al., 2019).

Original: In this system, network dynamics were used to generate a set of spike rates in simulated neurons.
Revised: In this system, network dynamics were used to generate a set of spike rates in simulated neurons.

Original: We then used a model of calcium dynamics and emissions noise to transform the spikes into synthetic ﬂuorescence data.
Revised: We then used a model of calcium dynamics and emissions noise to transform the spikes into synthetic ﬂuorescence data.

Original: We tested CaLFADS on data generated from two models of synthetic calcium dynamics: a simple linear model equivalent to an AR(1) process, and a nonlinear model in which an AR(1) process is transformed by a Hill function nonlinearity.
Revised: We tested CaLFADS on data generated from two models of synthetic calcium dynamics: a simple linear model equivalent to an AR(1) process, and a nonlinear model in which an AR(1) process is transformed by a Hill function nonlinearity.

Original: We then added white noise to the resulting traces.
Revised: We then added white noise to the resulting traces.

Original: This procedure is described in more detail in Appendix H.
Revised: This procedure is described in more detail in Appendix H.

Original: 3.1. M ODEL C OMPARISON
Revised: 3.1. M ODEL C OMPARISON

Original: We measured the performance of CaLFADS by computing R 2 goodness-of-ﬁt with the embedded
Revised: We measured the performance of CaLFADS by computing R 2 goodness-of-ﬁt with the embedded

Original: Lorenz attractor state variables.
Revised: Lorenz attractor state variables.

Original: We also measured the ability of CaLFADS to reconstruct other ground-truth variables in the synthetic data (spike counts, spike rates, and ﬂuorescence traces) using
Revised: We also measured the ability of CaLFADS to reconstruct other ground-truth variables in the synthetic data (spike counts, spike rates, and ﬂuorescence traces) using

Original: R 2 , but these were not as critical for assessing model performance.
Revised: R 2 , but these were not as critical for assessing model performance.

Original: CaLFADS performance was compared against two baselines.
Revised: CaLFADS performance was compared against two baselines.

Original: First, we use LFADS with a Gaussian likelihood observation model to account for ﬂuorescence (Gaussian-LFADS).
Revised: First, we use LFADS with a Gaussian likelihood observation model to account for ﬂuorescence (Gaussian-LFADS).

Original: Second, we consider the situation where spike counts are ﬁrst estimated separately using the OASIS deconvolution algorithm
Revised: Second, we consider the situation where spike counts are ﬁrst estimated separately using the OASIS deconvolution algorithm

Original: (Friedrich et al., 2017), a robust, popular, and computationally inexpensive method of spike extraction widely used by systems neuroscientists (Pachitariu et al., 2018; Evans et al., 2019).
Revised: (Friedrich et al., 2017), a robust, popular, and computationally inexpensive method of spike extraction widely used by systems neuroscientists (Pachitariu et al., 2018; Evans et al., 2019).

Original: LFADS is then used to infer the deeper-level dynamical system and reconstruct the spike rates from the deconvolved spike trains.
Revised: LFADS is then used to infer the deeper-level dynamical system and reconstruct the spike rates from the deconvolved spike trains.

Original: We refer to this approach as OASIS+LFADS throughout.
Revised: We refer to this approach as OASIS+LFADS throughout.

Original: Table 1 compares R 2 goodness-of-ﬁt in reconstructing ground-truth dynamic latent variables in held-out validation data.
Revised: Table 1 compares R 2 goodness-of-ﬁt in reconstructing ground-truth dynamic latent variables in held-out validation data.

Original: Table 1 shows CaLFADS is able to reconstruct the Lorenz attractor state for synthetic data with either linear or nonlinear synthetic calcium models (see Table 1 caption for signiﬁcance test results).
Revised: Table 1 shows CaLFADS is able to reconstruct the Lorenz attractor state for synthetic data with either linear or nonlinear synthetic calcium models (see Table 1 caption for signiﬁcance test results).

Original: Indeed, the inclusion of nonlinearities in calcium dynamics does not affect CaLFADS ability to reconstruct the Lorenz attractor state as much as it does with OASIS+LFADS.
Revised: Indeed, the inclusion of nonlinearities in calcium dynamics does not affect CaLFADS ability to reconstruct the Lorenz attractor state as much as it does with OASIS+LFADS.

Original: CaLFADS is also better able to reconstruct other network state variables compared to
Revised: CaLFADS is also better able to reconstruct other network state variables compared to

Original: OASIS+LFADS as can also be seen in Table 1.
Revised: OASIS+LFADS as can also be seen in Table 1.

Original: However, it is worth noting that high accuracy in reconstructing other network state variables does not appear to be necessary for accurate reconstruction of the underlying Lorenz attractor.
Revised: However, it is worth noting that high accuracy in reconstructing other network state variables does not appear to be necessary for accurate reconstruction of the underlying Lorenz attractor.

Original: This indicates that CaLFADS is better suited to handling uncertainty due to spike-timing and is capable of separating sources of slow dynamics to reconstruct the embedded latent space.
Revised: This indicates that CaLFADS is better suited to handling uncertainty due to spike-timing and is capable of separating sources of slow dynamics to reconstruct the embedded latent space.

Original: Example outputs of CaLFADS and OASIS+LFADS are shown in
Revised: Example outputs of CaLFADS and OASIS+LFADS are shown in

Original: Appendix I.1 and I.2.
Revised: Appendix I.1 and I.2.

Original: Note that CaLFADS reconstructs the Lorenz dynamics almost as well as LFADS does when applied to the spiking data.
Revised: Note that CaLFADS reconstructs the Lorenz dynamics almost as well as LFADS does when applied to the spiking data.

Original: Of course, it is to be expected that LFADS applied to true spiking data performs better than CaLFADS, since there is an additional source of observation noise from the generation of ﬂuorescence transients.
Revised: Of course, it is to be expected that LFADS applied to true spiking data performs better than CaLFADS, since there is an additional source of observation noise from the generation of ﬂuorescence transients.

Original: But, the fact we can get very close to the same level of perfor-
Revised: But, the fact we can get very close to the same level of perfor-

Original: [Table]
Revised: [Table]

Original: mance indicates that CaLFADS is effective at performing the same inference on calcium data that
Revised: mance indicates that CaLFADS is effective at performing the same inference on calcium data that

Original: LFADS performs on spiking data.
Revised: LFADS performs on spiking data.

Original: We speculate that the uncertainty over AR1 process parameters is overcome in CaLFADS by constraining the reconstructed spike counts with the latent dynamics to make the inference of the observed dynamics process “population-aware”, something which cannot be done when using LFADS applied to spike counts obtained by OASIS deconvolution.
Revised: We speculate that the uncertainty over AR1 process parameters is overcome in CaLFADS by constraining the reconstructed spike counts with the latent dynamics to make the inference of the observed dynamics process “population-aware”, something which cannot be done when using LFADS applied to spike counts obtained by OASIS deconvolution.

Original: This may mitigate errors in spike count reconstruction that occur by deconvolution due to the absence of information about whole-network dynamics.
Revised: This may mitigate errors in spike count reconstruction that occur by deconvolution due to the absence of information about whole-network dynamics.

Original: Thus, CaLFADS can use population-level dynamics when conducting the inference of network variables, and this helps it to separate out the shallower-level dynamics more accurately.
Revised: Thus, CaLFADS can use population-level dynamics when conducting the inference of network variables, and this helps it to separate out the shallower-level dynamics more accurately.

Original: 3. R OTATIONAL D YNAMICS IN M ONKEY M OTOR C ORTEX
Revised: 3. R OTATIONAL D YNAMICS IN M ONKEY M OTOR C ORTEX

Original: Next, we wanted to test CaLFADS on a real neural dataset to which LFADS has previously been applied, and successfully used to uncover meaningful latent dynamics.
Revised: Next, we wanted to test CaLFADS on a real neural dataset to which LFADS has previously been applied, and successfully used to uncover meaningful latent dynamics.

Original: Speciﬁcally, it has been shown previously that rotational dynamics underlie neuronal responses in monkey and human motor cortex during reaching behaviour (Churchland et al., 2012).
Revised: Speciﬁcally, it has been shown previously that rotational dynamics underlie neuronal responses in monkey and human motor cortex during reaching behaviour (Churchland et al., 2012).

Original: LFADS has been successful in uncovering these known rotational dynamics for single-trial spikes recorded from primary motor cortex (M1) and dorsal premotor cortex (PMd) in macaques (Pandarinath et al., 2018).
Revised: LFADS has been successful in uncovering these known rotational dynamics for single-trial spikes recorded from primary motor cortex (M1) and dorsal premotor cortex (PMd) in macaques (Pandarinath et al., 2018).

Original: To test whether CaLFADS could do the same, we took the original spiking data and converted it into semi-synthetic calcium traces, using the same model of calcium dynamics that was used for synthetic data generation in the previous sections (Fig 2A).
Revised: To test whether CaLFADS could do the same, we took the original spiking data and converted it into semi-synthetic calcium traces, using the same model of calcium dynamics that was used for synthetic data generation in the previous sections (Fig 2A).

Original: We used data from the monkey electrophysiology dataset previously described, along with the reaching task, by Churchland et al.
Revised: We used data from the monkey electrophysiology dataset previously described, along with the reaching task, by Churchland et al.

Original: (2012), which was kindly provided to us by the original authors.
Revised: (2012), which was kindly provided to us by the original authors.

Original: Brieﬂy, monkeys were trained to reach a target under 108 different reach conditions while multi-electrode recordings were made in M1 and PMd.
Revised: Brieﬂy, monkeys were trained to reach a target under 108 different reach conditions while multi-electrode recordings were made in M1 and PMd.

Original: Reaches started from a speciﬁed location on a screen, and monkeys were rewarded for correctly reaching toward a target while avoiding on-screen obstacles (Fig 2B).
Revised: Reaches started from a speciﬁed location on a screen, and monkeys were rewarded for correctly reaching toward a target while avoiding on-screen obstacles (Fig 2B).

Original: First, to replicate the previous LFADS results, we applied jPCA (Churchland et al., 2012) to condition-averaged ﬁring rates (trial averages for each of the 108 reach conditions), as well as singletrial ﬁring rates inferred from spike data using LFADS.
Revised: First, to replicate the previous LFADS results, we applied jPCA (Churchland et al., 2012) to condition-averaged ﬁring rates (trial averages for each of the 108 reach conditions), as well as singletrial ﬁring rates inferred from spike data using LFADS.

Original: jPCA is a dimensionality reduction technique that ﬁnds orthogonal projections capturing rotational dynamics that explain variability in ﬁring rates.
Revised: jPCA is a dimensionality reduction technique that ﬁnds orthogonal projections capturing rotational dynamics that explain variability in ﬁring rates.

Original: The original LFADS paper showed that rotational dynamics explained a large amount of the variance in ﬁring rates.
Revised: The original LFADS paper showed that rotational dynamics explained a large amount of the variance in ﬁring rates.

Original: As in the original papers, we identiﬁed both condition-averaged (Fig 2C – top) and single-trial (Fig 2C – bottom) rotational dynamics from ﬁring rates inferred by LFADS, which explained a large amount of the variance ( R 2 = 0 . 81 ), thereby replicating the original ﬁndings.
Revised: As in the original papers, we identiﬁed both condition-averaged (Fig 2C – top) and single-trial (Fig 2C – bottom) rotational dynamics from ﬁring rates inferred by LFADS, which explained a large amount of the variance ( R 2 = 0 . 81 ), thereby replicating the original ﬁndings.

Original: Then, we evaluated our CaLFADS model in uncovering the rotational dynamics of both conditionaveraged and single trials from ﬁring rates.
Revised: Then, we evaluated our CaLFADS model in uncovering the rotational dynamics of both conditionaveraged and single trials from ﬁring rates.

Original: As shown in Fig 2, our model also successfully uncovers rotational dynamics from calcium traces, explaining a large amount of variance ( R 2 = 0 . 78 ) for both that CaLFADS is capable of identifying latent dynamics in real neural data, similar to LFADS, even when the spiking data is transformed by calcium dynamics and emissions noise that the OASIS+LFADS approach was not as successful as CaLFADS in uncovering rotational dynamics, explaining only half the variance ( R 2 =
Revised: As shown in Fig 2, our model also successfully uncovers rotational dynamics from calcium traces, explaining a large amount of variance ( R 2 = 0 . 78 ) for both that CaLFADS is capable of identifying latent dynamics in real neural data, similar to LFADS, even when the spiking data is transformed by calcium dynamics and emissions noise that the OASIS+LFADS approach was not as successful as CaLFADS in uncovering rotational dynamics, explaining only half the variance ( R 2 =

Original: 53 , Fig 2E).
Revised: 53 , Fig 2E).

Original: This result corroborates the importance of the hierarchical modelling of calcium and neuronal dynamics in CaLFADS.
Revised: This result corroborates the importance of the hierarchical modelling of calcium and neuronal dynamics in CaLFADS.

Original: 3. U NEXPECTED S TIMULUS D ETECTION IN M OUSE P RIMARY V ISUAL C ORTEX
Revised: 3. U NEXPECTED S TIMULUS D ETECTION IN M OUSE P RIMARY V ISUAL C ORTEX

Original: Finally we wanted to test CaLFADS on an entirely new calcium imaging dataset, to determine whether CaLFADS can infer dynamic computational factors that carry information about relevant features of the outside world.
Revised: Finally we wanted to test CaLFADS on an entirely new calcium imaging dataset, to determine whether CaLFADS can infer dynamic computational factors that carry information about relevant features of the outside world.

Original: We chose to analyse data from mouse primary visual cortex (VisP), a region widely studied in systems neuroscience research where calcium imaging is a standard tool.
Revised: We chose to analyse data from mouse primary visual cortex (VisP), a region widely studied in systems neuroscience research where calcium imaging is a standard tool.

Original: There is evidence that the visual cortex of mammals performs a predictive function, anticipating expected stimuli (Dayan et al., 1995;
Revised: There is evidence that the visual cortex of mammals performs a predictive function, anticipating expected stimuli (Dayan et al., 1995;

Original: Rao & Ballard, 1999; Fiser et al., 2016; Leinweber et al., 2017).
Revised: Rao & Ballard, 1999; Fiser et al., 2016; Leinweber et al., 2017).

Original: As a result, unexpected stimuli can induce perturbations in network dynamics (Fiser et al., 2016;
Revised: As a result, unexpected stimuli can induce perturbations in network dynamics (Fiser et al., 2016;

Original: Leinweber et al., 2017).
Revised: Leinweber et al., 2017).

Original: Thus, we wanted to determine whether CaLFADS could infer dynamic computational factors that carry information distinguishing unexpected from expected visual stimuli.
Revised: Thus, we wanted to determine whether CaLFADS could infer dynamic computational factors that carry information distinguishing unexpected from expected visual stimuli.

Original: To this end, we trained CaLFADS on calcium imaging data from the mouse visual cortex collected by the Allen Institute for Brain Science (for a detailed description, see Appendix J.1).
Revised: To this end, we trained CaLFADS on calcium imaging data from the mouse visual cortex collected by the Allen Institute for Brain Science (for a detailed description, see Appendix J.1).

Original: While awake behaving mice were presented with visual stimuli on a screen (Fig 3A), calcium ﬂuorescence responses in cortical layer 2/3 of VisP were recorded using 2-photon microscopy (Fig 3B).
Revised: While awake behaving mice were presented with visual stimuli on a screen (Fig 3A), calcium ﬂuorescence responses in cortical layer 2/3 of VisP were recorded using 2-photon microscopy (Fig 3B).

Original: The mice were familiarised with sequences of stimulus frames that followed simple probabilistic rules.
Revised: The mice were familiarised with sequences of stimulus frames that followed simple probabilistic rules.

Original: Brieﬂy, each sequence consisted of four randomised Gabor pattern frames followed by a grey screen (A, B,
Revised: Brieﬂy, each sequence consisted of four randomised Gabor pattern frames followed by a grey screen (A, B,

Original: C, D, grey) with orientations drawn for each trial from the same distribution.
Revised: C, D, grey) with orientations drawn for each trial from the same distribution.

Original: After familiarization with this sequence, the expected D frame was replaced in 10% of trials by a Gabor pattern (E) with orthogonal orientations (Fig 3C).
Revised: After familiarization with this sequence, the expected D frame was replaced in 10% of trials by a Gabor pattern (E) with orthogonal orientations (Fig 3C).

Original: CaLFADS inferred smooth latent factors that did not show signs of over-ﬁtting to noise (Fig S6).
Revised: CaLFADS inferred smooth latent factors that did not show signs of over-ﬁtting to noise (Appendix J.2, Fig S7).

Original: To determine if salient information about these unexpected stimulus features was present in the inferred factors from CaLFADS, we trained a non-linear decoder to classify whether the inferred factors came from a trial with an expected D frame or an unexpected E frame (for details, see Appendix J.2).
Revised: To determine if salient information about these unexpected stimulus features was present in the inferred factors from CaLFADS, we trained a nonlinear decoder to classify whether the inferred factors came from a trial with an expected D frame or an unexpected E frame (for details, see Appendix J.3).

Original: We also trained the non-linear decoder on factors inferred by OASIS+LFADS, and on ﬂuorescence traces reduced in dimensionality with principal components analysis.
Revised: We also trained the non-linear decoder on factors inferred by OASIS+LFADS, and on ﬂuorescence traces reduced in dimensionality with prin- cipal components analysis.

Original: To compare the models fairly, despite the imbalance between expected and unexpected trial frequencies, we measured average recall.
Revised: To compare the models fairly, despite the imbalance between expected and unexpected trial frequencies, we measured average recall.

Original: We ﬁnd that using the CaLFADS factors leads to the highest performance on stimulus-trial identity decoding, with the average recall being signiﬁcantly higher than when OASIS+LFADS factors are used (Fig 3D).
Revised: We ﬁnd that using the CaLFADS factors leads to the highest performance on stimulus-trial identity decoding, with the average recall being signiﬁcantly higher than when OASIS+LFADS factors are used (Fig 3D).

Original: This indicates that CaLFADS infers latent dynamics from real visual cortex calcium imaging data that reﬂect the predictability of visual stimuli, corroborating the ability of CaLFADS to infer meaningful dynamic factors in real data.
Revised: This indicates that CaLFADS infers latent dynamics from real visual cortex calcium imaging data that reﬂect the predictability of visual stimuli, corroborating the ability of CaLFADS to infer meaningful dynamic factors in real data.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper we presented CaLFADS, a hierarchical recurrent variational autoencoder model capable of reconstructing latent computational dynamics.
Revised: In this paper we presented CaLFADS, a hierarchical recurrent variational autoencoder model capable of reconstructing latent computational dynamics.

Original: We conﬁrmed CaLFADS’ ability to reconstruct known underlying dynamics using synthetic datasets where ground-truth was known.
Revised: We conﬁrmed CaLFADS’ ability to reconstruct known underlying dynamics using synthetic datasets where ground-truth was known.

Original: We also showed that CaLFADS is able to infer sensorimotor dynamics from real neural recordings.
Revised: We also showed that CaLFADS is able to infer sensorimotor dynamics from real neural recordings.

Original: This indicates that CaLFADS is a promising method for analyzing calcium imaging data in neuroscience.
Revised: This indicates that CaLFADS is a promising method for analyzing calcium imaging data in neuroscience.

Original: There are two key advantages of CaLFADS over the use of deconvolution of calcium traces followed by application of LFADS.
Revised: There are two key advantages of CaLFADS over the use of deconvolution of calcium traces followed by application of LFADS.

Original: The ﬁrst is that we can obtain measures of the uncertainty in both the latent dynamics, and the latent spike counts.
Revised: The ﬁrst is that we can obtain measures of the uncertainty in both the latent dynamics, and the latent spike counts.

Original: The second is that CaLFADS performs better than
Revised: The second is that CaLFADS performs better than

Original: OASIS+LFADS when nonlinearities in calcium dynamics are present, as is the case in real experimental data where there are many sources of calcium inﬂux related to spikes.
Revised: OASIS+LFADS when nonlinearities in calcium dynamics are present, as is the case in real experimental data where there are many sources of calcium inﬂux related to spikes.

Original: We designed CaLFADS to ﬁt into a much broader class of artiﬁcial neural network models, namely sequential variational ladder autoencoders , in which different layers of recurrent neural networks can be used to infer and generate different layers of a hierarchical dynamical system.
Revised: We designed CaLFADS to ﬁt into a much broader class of artiﬁcial neural network models, namely sequential variational ladder autoencoders , in which different layers of recurrent neural networks can be used to infer and generate different layers of a hierarchical dynamical system.

Original: In this sense, this class of models is modular and composable, meaning it could be readily adapted to other domains.
Revised: In this sense, this class of models is modular and composable, meaning it could be readily adapted to other domains.

Original: For example, it should be possible to replace LFADS as the deeper-level dynamical system model with any other differentiable model.
Revised: For example, it should be possible to replace LFADS as the deeper-level dynamical system model with any other differentiable model.

Original: Likewise the same is true for our AR1 based model of calcium dynamics: we could replace the calcium dynamics model with other models as required by the experimental set-up.
Revised: Likewise the same is true for our AR1 based model of calcium dynamics: we could replace the calcium dynamics model with other models as required by the experimental set-up.

Original: Furthermore, there is no need to stop at two layers in the hierarchy; the brain is comprised of many interconnected recurrent neural networks sending long range signals to one another, and it should be possible to add in additional modules to capture additional hierarchies and dynamics within the brain.
Revised: Furthermore, there is no need to stop at two layers in the hierarchy; the brain is comprised of many interconnected recurrent neural networks sending long range signals to one another, and it should be possible to add in additional modules to capture additional hierarchies and dynamics within the brain.

Original: In summary, CaLFADS is a new, open source tool for inferring hierarchical dynamics from calcium imaging data that also has great potential for being modiﬁed and applied to other data modalities.
Revised: In summary, CaLFADS is a new, open source tool for inferring hierarchical dynamics from calcium imaging data that also has great potential for being modiﬁed and applied to other data modalities.

Original: This could be of real beneﬁt to the thousands of neuroscience laboratories around the world conducting calcium imaging experiments.
Revised: This could be of real beneﬁt to the thousands of neuroscience laboratories around the world conducting calcium imaging experiments.

Original: A V ARIABLE G LOSSARY
Revised: A V ARIABLE G LOSSARY

Original: [Table]
Revised: [Table]

Original: B R ELATED W ORK AND C ONTRIBUTION
Revised: B R ELATED W ORK AND C ONTRIBUTION

Original: This work brings together two important strands of computational neuroscience research that have been of interest to the machine learning community, namely probabilistic latent factor models of neural dynamics and inference of spikes in calcium ﬂuorescence traces.
Revised: This work brings together two important strands of computational neuroscience research that have been of interest to the machine learning community, namely probabilistic latent factor models of neural dynamics and inference of spikes in calcium ﬂuorescence traces.

Original: We solve these two problems as a hierarchical inference problem using variational ladder autoencoders.
Revised: We solve these two problems as a hierarchical inference problem using variational ladder autoencoders.

Original: We give a brief introduction to variational ladder autoencoders in Appendix C.
Revised: We give a brief introduction to variational ladder autoencoders in Appendix C.

Original: LFADS is a power latent factor model of neural dynamics proposed by Pandarinath et al.
Revised: LFADS is a power latent factor model of neural dynamics proposed by Pandarinath et al.

Original: It is an extension of the sequential variational autoencoder DRAW (Gregor et al., 2015) applied to spiking data that adds an additional RNN to the encoder architecture to infer unknown initial conditions to a generator RNN.
Revised: It is an extension of the sequential variational autoencoder DRAW (Gregor et al., 2015) applied to spiking data that adds an additional RNN to the encoder architecture to infer unknown initial conditions to a generator RNN.

Original: This enables interpretation of the generative model as a nonlinear dynamical system that generates observed spikes.
Revised: This enables interpretation of the generative model as a nonlinear dynamical system that generates observed spikes.

Original: Low dimensional projections of the generator hidden state (latent factors) were found to be capable of decoding single-trial reaching movements with very high accuracy.
Revised: Low dimensional projections of the generator hidden state (latent factors) were found to be capable of decoding single-trial reaching movements with very high accuracy.

Original: This was a hugely inﬂuential and important contribution to the ﬁeld of neuroscience as it demonstrated a means to ﬁnd latent dynamical systems that solved computational tasks required for executing behaviour.
Revised: This was a hugely inﬂuential and important contribution to the ﬁeld of neuroscience as it demonstrated a means to ﬁnd latent dynamical systems that solved computational tasks required for executing behaviour.

Original: However, LFADS could not be applied to neural data where the spiking was itself unknown, as in two-photon calcium imaging.
Revised: However, LFADS could not be applied to neural data where the spiking was itself unknown, as in two-photon calcium imaging.

Original: DeepSpike by Speiser et al.
Revised: DeepSpike by Speiser et al.

Original: (2017) is a variational autoencoder adapted from MLSpike, a probabilistic model of calcium ﬂuorescence recordings (Deneux et al., 2016).
Revised: (2017) is a variational autoencoder adapted from MLSpike, a probabilistic model of calcium ﬂuorescence recordings (Deneux et al., 2016).

Original: It was a contribution to the
Revised: It was a contribution to the

Original: SpikeFinder competition (Berens et al., 2018) in which participants were given the task of inferring an unknown spike train from calcium ﬂuorescence recordings.
Revised: SpikeFinder competition (Berens et al., 2018) in which participants were given the task of inferring an unknown spike train from calcium ﬂuorescence recordings.

Original: Unlike in most two-photon recordings, these data also included ground-truth whole-cell electrophysiology, i.e., ground-truth spike trains.
Revised: Unlike in most two-photon recordings, these data also included ground-truth whole-cell electrophysiology, i.e., ground-truth spike trains.

Original: While other techniques had higher performance, it was the ﬁrst attempt at solving this problem using variational inference.
Revised: While other techniques had higher performance, it was the ﬁrst attempt at solving this problem using variational inference.

Original: One issue with this model is that it used a Bernoulli distribution to model the latent spike train.
Revised: One issue with this model is that it used a Bernoulli distribution to model the latent spike train.

Original: Since this is not reparameterisable, Speiser et al.
Revised: Since this is not reparameterisable, Speiser et al.

Original: (2017) needed to use a high variance Monte Carlo objective to train the model.
Revised: (2017) needed to use a high variance Monte Carlo objective to train the model.

Original: The use of this objective means it would not scale well to 2-photon measurements from many cells.
Revised: The use of this objective means it would not scale well to 2-photon measurements from many cells.

Original: Furthermore, such a model does not capture how neurons can spike multiple times within the time-bin of a typical two-photon imaging experiment.
Revised: Furthermore, such a model does not capture how neurons can spike multiple times within the time-bin of a typical two-photon imaging experiment.

Original: We note some similarities between our generative model and Aitchison et al.
Revised: We note some similarities between our generative model and Aitchison et al.

Original: A key difference here is that we try to ﬁnd nonlinear latent dynamics, whereas their model comprises a linear dynamical system in the latent space.
Revised: A key difference here is that we try to ﬁnd nonlinear latent dynamics, whereas their model comprises a linear dynamical system in the latent space.

Original: Furthermore, theirs is a fully Bayesian approach, whereas
Revised: Furthermore, theirs is a fully Bayesian approach, whereas

Original: CaLFADS has many more deterministic components.
Revised: CaLFADS has many more deterministic components.

Original: We also note the capabilities of VIND (Hernandez et al., 2020) in inferring non-linear latent dynamics in wide-ﬁeld calcium imaging data, which has insufﬁcient spatial resolution to observe individual cells.
Revised: We also note the capabilities of VIND (Hernandez et al., 2020) in inferring non-linear latent dynamics in wide-ﬁeld calcium imaging data, which has insufﬁcient spatial resolution to observe individual cells.

Original: Hernandez et al.
Revised: Hernandez et al.

Original: (2020) propose a variational inference technique in which latent dynamics are modelled by a locally linear dynamical system and ﬁnd informative latent trajectories in the recorded smoothed population activity.
Revised: (2020) propose a variational inference technique in which latent dynamics are modelled by a locally linear dynamical system and ﬁnd informative latent trajectories in the recorded smoothed population activity.

Original: We note however that this approach is not used to infer latent dynamics in two-photon recordings, where it would be necessary to disentangle calcium dynamics from population dynamics.
Revised: We note however that this approach is not used to infer latent dynamics in two-photon recordings, where it would be necessary to disentangle calcium dynamics from population dynamics.

Original: Finally, the variational autoencoder LeMoNADe proposed by Kirschbaum et al.
Revised: Finally, the variational autoencoder LeMoNADe proposed by Kirschbaum et al.

Original: (2019) similarly proposes an end-to-end system of inferring latent activities from calcium imaging.
Revised: (2019) similarly proposes an end-to-end system of inferring latent activities from calcium imaging.

Original: In contrast to
Revised: In contrast to

Original: CaLFADS, they are attempting to infer the presence of neuronal ensembles, sets of cells grouped together by their activities.
Revised: CaLFADS, they are attempting to infer the presence of neuronal ensembles, sets of cells grouped together by their activities.

Original: While this is a related goal to inferring a latent dynamical system, there is no requirement for the latent factors to be as interpretable as in LeMoNADe.
Revised: While this is a related goal to inferring a latent dynamical system, there is no requirement for the latent factors to be as interpretable as in LeMoNADe.

Original: Furthermore, the neuronal ensembles identiﬁed by LeMoNADe were not demonstrated to help solve relevant behavioural tasks.
Revised: Furthermore, the neuronal ensembles identiﬁed by LeMoNADe were not demonstrated to help solve relevant behavioural tasks.

Original: However, LeMoNADe were the ﬁrst to demonstrate how neuronal activities could be modelled by a continuous relaxation to the Bernoulli distribution using the Gumbel-Softmax trick.
Revised: However, LeMoNADe were the ﬁrst to demonstrate how neuronal activities could be modelled by a continuous relaxation to the Bernoulli distribution using the Gumbel-Softmax trick.

Original: CaLFADS ties together this related work.
Revised: CaLFADS ties together this related work.

Original: We propose a hierarchical variational autoencoder capable of inferring latent dynamic factors in calcium imaging data by separating shallower-level calcium dynamics and deeper-level computational dynamics.
Revised: We propose a hierarchical variational autoencoder capable of inferring latent dynamic factors in calcium imaging data by separating shallower-level calcium dynamics and deeper-level computational dynamics.

Original: We propose a novel continuous relaxation to spike counts that allows an arbitrary number of spikes to be counted within the typically low sampling rate of calcium ﬂuorescence recordings (
Revised: We propose a novel continuous relaxation to spike counts that allows an arbitrary number of spikes to be counted within the typically low sampling rate of calcium ﬂuorescence recordings (

Original: Hz)
Revised: Hz)

Original: We also propose a novel KL annealing technique for allowing hierarchical latent dynamics to be inferred
Revised: We also propose a novel KL annealing technique for allowing hierarchical latent dynamics to be inferred

Original: C I NTRODUCTION TO V ARIATIONAL L ADDER A UTOENCODERS
Revised: C I NTRODUCTION TO V ARIATIONAL L ADDER A UTOENCODERS

Original: Variational Ladder Autoencoders were proposed as a way to ensure that higher- and lower-level features could be stored in different parts of the latent state.
Revised: Variational Ladder Autoencoders were proposed as a way to ensure that higher- and lower-level features could be stored in different parts of the latent state.

Original: They propose a ladder architecture in which more expressive networks sit deeper in the hierarchy and no hierarchy is assumed among latent variables directly.
Revised: They propose a ladder architecture in which more expressive networks sit deeper in the hierarchy and no hierarchy is assumed among latent variables directly.

Original: For layers 1 : L in the hierarchy
Revised: For layers 1 : L in the hierarchy

Original: [Equation]
Revised: [Equation]

Original: where h 0 = x , and g l and f l are neural networks with increasing degrees of expressivity as l increases.
Revised: where h 0 = x , and g l and f l are neural networks with increasing degrees of expressivity as l increases.

Original: This encourages more abstract features to be represented in certain portions of the latent state.
Revised: This encourages more abstract features to be represented in certain portions of the latent state.

Original: This is in contrast to traditional hierarchical autoencoders that assume a Markovian structure in the latent variables
Revised: This is in contrast to traditional hierarchical autoencoders that assume a Markovian structure in the latent variables

Original: [Equation]
Revised: [Equation]

Original: The key insight from (Zhao et al., 2017) is that under idealised assumptions for sampling and optimization of the ELBO, the ELBO for hierarchical VAEs is optimised by minimizing only
Revised: The key insight from (Zhao et al., 2017) is that under idealised assumptions for sampling and optimization of the ELBO, the ELBO for hierarchical VAEs is optimised by minimizing only

Original: D KL ( Q ( z 1 | x ) || P ( z 1 | x )) .
Revised: D KL ( Q ( z 1 | x ) || P ( z 1 | x )) .

Original: This means that traditional hierarchical VAEs will load all representations onto the lowest level of the hierarchy.
Revised: This means that traditional hierarchical VAEs will load all representations onto the lowest level of the hierarchy.

Original: In contrast, by splitting the latent code by abstractness and parameterising the factorised posterior distribution with increasingly expressive functions, the VLAE can be trained with the simple ELBO formulation used in non-hierarchical VAEs.
Revised: In contrast, by splitting the latent code by abstractness and parameterising the factorised posterior distribution with increasingly expressive functions, the VLAE can be trained with the simple ELBO formulation used in non-hierarchical VAEs.

Original: D C A LFADS PROBABILISTIC MODEL
Revised: D C A LFADS PROBABILISTIC MODEL

Original: Here we outline the assumed probabilistic model for CaLFADS.
Revised: Here we outline the assumed probabilistic model for CaLFADS.

Original: In doing so, we will brieﬂy review the model of LFADS (Pandarinath et al., 2018) and the extensions that make it possible to infer dynamic latent variables from calcium ﬂuorescence traces.
Revised: In doing so, we will brieﬂy review the model of LFADS (Pandarinath et al., 2018) and the extensions that make it possible to infer dynamic latent variables from calcium ﬂuorescence traces.

Original: D. LFADS G ENERATIVE M ODEL
Revised: D. LFADS G ENERATIVE M ODEL

Original: LFADS assumes that neural activity x is generated from a low dimensional non-linear dynamical system with unknown initial conditions subjected to unknown control inputs.
Revised: LFADS assumes that neural activity x is generated from a low dimensional non-linear dynamical system with unknown initial conditions subjected to unknown control inputs.

Original: The state of the lowdimensional non-linear dynamical system is modelled by the hidden state g t of a Gated Recurrent
Revised: The state of the lowdimensional non-linear dynamical system is modelled by the hidden state g t of a Gated Recurrent

Original: Unit receiving control inputs u t .
Revised: Unit receiving control inputs u t .

Original: The prior for the initial conditions g 0 is assumed to be normal with mean µ g 0 and variance σ 2 g 0 .
Revised: The prior for the initial conditions g 0 is assumed to be normal with mean µ g 0 and variance σ 2 g 0 .

Original: External inputs are assumed to have an autoregressive process prior with process mean µ u , process variance σ 2 u , and time constant τ x ,
Revised: External inputs are assumed to have an autoregressive process prior with process mean µ u , process variance σ 2 u , and time constant τ x ,

Original: [Equation]
Revised: [Equation]

Original: The hidden states of the GRU are then projected to a low-dimensional set of factors f t
Revised: The hidden states of the GRU are then projected to a low-dimensional set of factors f t

Original: [Equation]
Revised: [Equation]

Original: In Pandarinath et al.
Revised: In Pandarinath et al.

Original: , the neural activity being modelled was single-trial spikes extracted from electrophysiological data in macaque motor cortex.
Revised: , the neural activity being modelled was single-trial spikes extracted from electrophysiological data in macaque motor cortex.

Original: LFADS assumes a simple observation model for spiking data where the spike count observed in a given time-bin x t follows an independent Poisson process parameterised by the instantaneous ﬁring rate
Revised: LFADS assumes a simple observation model for spiking data where the spike count observed in a given time-bin x t follows an independent Poisson process parameterised by the instantaneous ﬁring rate

Original: λ t .
Revised: λ t .

Original: To obtain the instantaneous ﬁring rate, low-dimensional factors f t are linearly projected to the dimensionality of the data and a non-linearity enforcing positivity is applied, in this case the exponential function
Revised: To obtain the instantaneous ﬁring rate, low-dimensional factors f t are linearly projected to the dimensionality of the data and a non-linearity enforcing positivity is applied, in this case the exponential function

Original: [Equation]
Revised: [Equation]

Original: D. C ALCIUM F LUORESCENCE G ENERATIVE M ODEL
Revised: D. C ALCIUM F LUORESCENCE G ENERATIVE M ODEL

Original: Before describing how the low dimensional dynamic factors are used to generate calcium ﬂuorescence data, we ﬁrst describe our probabilistic model for calcium ﬂuorescence transients.
Revised: Before describing how the low dimensional dynamic factors are used to generate calcium ﬂuorescence data, we ﬁrst describe our probabilistic model for calcium ﬂuorescence transients.

Original: We assume that calcium ﬂuorescence transients are generated by an autoregressive process
Revised: We assume that calcium ﬂuorescence transients are generated by an autoregressive process

Original: c t occluded by white emissions noise (cid:15) and perturbed by an unknown spike train s t .
Revised: c t occluded by white emissions noise (cid:15) and perturbed by an unknown spike train s t .

Original: Probabilistic models of unknown spike counts in calcium ﬂuorescence data typically assume a
Revised: Probabilistic models of unknown spike counts in calcium ﬂuorescence data typically assume a

Original: Bernoulli distribution, or some continuous approximation thereof (Speiser et al., 2017).
Revised: Bernoulli distribution, or some continuous approximation thereof (Speiser et al., 2017).

Original: This is somewhat problematic given that ﬂuorescence is sampled at a rate of 10-30 Hz, leaving time-bins large enough to observe more than one spike per time-bin.
Revised: This is somewhat problematic given that ﬂuorescence is sampled at a rate of 10-30 Hz, leaving time-bins large enough to observe more than one spike per time-bin.

Original: Recently, Wei et al. (2019a) demonstrated that spike counts inferred by the deconvolution algorithm OASIS (Friedrich et al., 2017) could be closely modelled by a zero-inﬂated gamma model.
Revised: Recently, Wei et al. (2019a) demonstrated that spike counts inferred by the deconvolution algorithm OASIS (Friedrich et al., 2017) could be closely modelled by a zero-inﬂated gamma model.

Original: Similar to this, we modelled spike counts using a similar continuous approximation
Revised: Similar to this, we modelled spike counts using a similar continuous approximation

Original: [Equation]
Revised: [Equation]

Original: This approximation is easily reparameterisable since we can sample from a lognormal distribution by transforming samples from a normal distribution with an exponential function.
Revised: This approximation is easily reparameterisable since we can sample from a lognormal distribution by transforming samples from a normal distribution with an exponential function.

Original: The ReLU function and subtraction of 1 from samples ensures that many counts are zero.
Revised: The ReLU function and subtraction of 1 from samples ensures that many counts are zero.

Original: In contrast, since the zeroinﬂated gamma model which is a mixture distribution, it would require modelling a hierarchical discrete-continuous latent variable which is not as simply reparameterisable for backpropagation.
Revised: In contrast, since the zeroinﬂated gamma model which is a mixture distribution, it would require modelling a hierarchical discrete-continuous latent variable which is not as simply reparameterisable for backpropagation.

Original: Spike counts are then used to perturb an autoregressive process modelling the slow decay of calcium ﬂuorescence after a spike.
Revised: Spike counts are then used to perturb an autoregressive process modelling the slow decay of calcium ﬂuorescence after a spike.

Original: A simple autoregressive model for calcium ﬂuorescence is the AR(1) process.
Revised: A simple autoregressive model for calcium ﬂuorescence is the AR(1) process.

Original: With discretised time, this can be modelled with Euler updates
Revised: With discretised time, this can be modelled with Euler updates

Original: [Equation]
Revised: [Equation]

Original: where β = ∆ t τ c , with ∆ t the size of time bin and τ c the decay of calcium ﬂuorescence, and with ∆ t < τ c .
Revised: where β = ∆ t τ c , with ∆ t the size of time bin and τ c the decay of calcium ﬂuorescence, and with ∆ t < τ c .

Original: α is a constant gain term controlling the effect of spikes on ﬂuorescence.
Revised: α is a constant gain term controlling the effect of spikes on ﬂuorescence.

Original: A nonlinear function modelling dye kinetics d ( · ) can then be applied.
Revised: A nonlinear function modelling dye kinetics d ( · ) can then be applied.

Original: For these experiments we have not done so, but it would be a relatively simple extension.
Revised: For these experiments we have not done so, but it would be a relatively simple extension.

Original: As such, the observation model for ﬂuorescence F becomes
Revised: As such, the observation model for ﬂuorescence F becomes

Original: [Equation]
Revised: [Equation]

Original: where in our case, d ( c t )
Revised: where in our case, d ( c t )

Original: =
Revised: =

Original: c t .
Revised: c t .

Original: D. C A LFADS GENERATIVE MODEL
Revised: D. C A LFADS GENERATIVE MODEL

Original: To tie LFADS and our observation model for calcium ﬂuorescence together, we will adjust the notation slightly to make the hierarchical positioning of latent variables clearer.
Revised: To tie LFADS and our observation model for calcium ﬂuorescence together, we will adjust the notation slightly to make the hierarchical positioning of latent variables clearer.

Original: The variables of the deeper-level model of LFADS are denoted by the subscript l = 2 , whereas the variables of the shallower-level model of observed calcium ﬂuorescence are denoted by the subscript l = 1 .
Revised: The variables of the deeper-level model of LFADS are denoted by the subscript l = 2 , whereas the variables of the shallower-level model of observed calcium ﬂuorescence are denoted by the subscript l = 1 .

Original: For consistency and notation simplicity, we denote variables that are the state of a dynamical system by g l,t and inputs to the dynamical system as u l,t .
Revised: For consistency and notation simplicity, we denote variables that are the state of a dynamical system by g l,t and inputs to the dynamical system as u l,t .

Original: We observe neural activity x in the form of single-trial calcium ﬂuorescence traces.
Revised: We observe neural activity x in the form of single-trial calcium ﬂuorescence traces.

Original: We assume that ﬂuorescence follows a simple autoregressive process g l = 1 perturbed by an unknown set of spike counts u l = 1 and subject to additive white emissions noise.
Revised: We assume that ﬂuorescence follows a simple autoregressive process g l = 1 perturbed by an unknown set of spike counts u l = 1 and subject to additive white emissions noise.

Original: We also assume that spike counts u l =are inﬂuenced by the state of a nonlinear dynamical system with unknown initial conditions
Revised: We also assume that spike counts u l =are inﬂuenced by the state of a nonlinear dynamical system with unknown initial conditions

Original: g l = 2 , t =and unknown inputs u l = 2 .
Revised: g l = 2 , t =and unknown inputs u l = 2 .

Original: As in LFADS, the nonlinear dynamical system is modelled by a GRU.
Revised: As in LFADS, the nonlinear dynamical system is modelled by a GRU.

Original: The hidden state of the GRU g l =2 ,t is projected to a low-dimensional set of factors f t
Revised: The hidden state of the GRU g l =2 ,t is projected to a low-dimensional set of factors f t

Original: [Equation]
Revised: [Equation]

Original: For spike counts to be informed by the dynamic factors f t , we sample spike counts according to
Revised: For spike counts to be informed by the dynamic factors f t , we sample spike counts according to

Original: [Equation]
Revised: [Equation]

Original: using the continuous approximation to spike counts described in section D.2.
Revised: using the continuous approximation to spike counts described in section D.2.

Original: Although this factorization separates the generation of approximate spikes counts from the dynamic latent factors, for the formulation of the loss function we treat u 1 , t as if it is generated from a homogeneous Poisson process with intensity λ t = exp( W λ f t + b λ ), i.e.,
Revised: Although this factorization separates the generation of approximate spikes counts from the dynamic latent factors, for the formulation of the loss function we treat u 1 , t as if it is generated from a homogeneous Poisson process with intensity λ t = exp( W λ f t + b λ ), i.e.,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We then use these approximated spikes counts to perturb the state g l =1 ,t of an AR(1) process
Revised: We then use these approximated spikes counts to perturb the state g l =1 ,t of an AR(1) process

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The likelihood of the observed ﬂuorescence in a given time bin x t is then modelled as normal random variable with a mean dependent on the state of the AR(1) process.
Revised: The likelihood of the observed ﬂuorescence in a given time bin x t is then modelled as normal random variable with a mean dependent on the state of the AR(1) process.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where again d ( · ) is an optional function modelling dye kinetics.
Revised: where again d ( · ) is an optional function modelling dye kinetics.

Original: In our case d ( g 1 , t ) =
Revised: In our case d ( g 1 , t ) =

Original: g 1 , t .
Revised: g 1 , t .

Original: D. C A LFADS INFERENCE NETWORK
Revised: D. C A LFADS INFERENCE NETWORK

Original: The inference network of CaLFADS parameterises the approximate posterior distributions
Revised: The inference network of CaLFADS parameterises the approximate posterior distributions

Original: Q ( g 2 , 0 | x ) , Q ( u 2 ,t | x ) and Q ( z | x ) .
Revised: Q ( g 2 , 0 | x ) , Q ( u 2 ,t | x ) and Q ( z | x ) .

Original: As per the ladder architecture (Zhao et al., 2017) these posterior distributions take the form
Revised: As per the ladder architecture (Zhao et al., 2017) these posterior distributions take the form

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 26) where p ( · ) , q ( · ) , r ( · ) are neural networks parameterised by φ , θ , ψ respectively.
Revised: 26) where p ( · ) , q ( · ) , r ( · ) are neural networks parameterised by φ , θ , ψ respectively.

Original: The combination of latent space factorization and parameter sharing ensures that dynamics can be disentangled effectively.
Revised: The combination of latent space factorization and parameter sharing ensures that dynamics can be disentangled effectively.

Original: As in LFADS, the approximate posterior distribution over the initial conditions to the nonlinear dynamical system Q ( g 2 , 0 ) is parameterised separately using a bidirectional Gated Recurrent Unit
Revised: As in LFADS, the approximate posterior distribution over the initial conditions to the nonlinear dynamical system Q ( g 2 , 0 ) is parameterised separately using a bidirectional Gated Recurrent Unit

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 29) where h g 2 , 0 is the ﬁnal hidden state of a bidirectional Gated Recurrent unit
Revised: 29) where h g 2 , 0 is the ﬁnal hidden state of a bidirectional Gated Recurrent unit

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where h g 2 , 0 ,fwd t is the state of a GRU running forward sequentially over the input, and h g 2 , 0 ,bwd t is the state of a GRU running backward sequentially over the data.
Revised: where h g 2 , 0 ,fwd t is the state of a GRU running forward sequentially over the input, and h g 2 , 0 ,bwd t is the state of a GRU running backward sequentially over the data.

Original: The functions f φ ( · ) , g θ ( · ) , h ψ ( · ) are modelled by pairs of recurrent neural networks.
Revised: The functions f φ ( · ) , g θ ( · ) , h ψ ( · ) are modelled by pairs of recurrent neural networks.

Original: As in LFADS, we use bidirectional GRUs followed by unidirectional controller GRUs receiving representations from the generative network.
Revised: As in LFADS, we use bidirectional GRUs followed by unidirectional controller GRUs receiving representations from the generative network.

Original: The bidirectional GRU is deﬁned as before
Revised: The bidirectional GRU is deﬁned as before

Original: [Equation]
Revised: [Equation]

Original: From this point the inference model splits into the two levels of the hierarchy.
Revised: From this point the inference model splits into the two levels of the hierarchy.

Original: The hidden state h ut is passed through the a controller GRU which takes feedback from the generated samples
Revised: The hidden state h ut is passed through the a controller GRU which takes feedback from the generated samples

Original: [Equation]
Revised: [Equation]

Original: The states of the controller GRU are then linearly transformed onto the parameters of a normal distribution
Revised: The states of the controller GRU are then linearly transformed onto the parameters of a normal distribution

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: D. C A LFADS LOSS FUNCTION
Revised: D. C A LFADS LOSS FUNCTION

Original: To construct the cost function of CaLFADS, we start from the evidence lower bound (ELBO)
Revised: To construct the cost function of CaLFADS, we start from the evidence lower bound (ELBO)

Original: [Equation]
Revised: [Equation]

Original: where x denotes the observed data and ˆx denotes the reconstructed data.
Revised: where x denotes the observed data and ˆx denotes the reconstructed data.

Original: With our model, this factorises as
Revised: With our model, this factorises as

Original: [Equation]
Revised: [Equation]

Original: Since u 1 ,t is a continuous random variable that we are treating as if it came from an independent, homogeneous Poisson process with parameter λ t , we approximate the log-likelihood with
Revised: Since u 1 ,t is a continuous random variable that we are treating as if it came from an independent, homogeneous Poisson process with parameter λ t , we approximate the log-likelihood with

Original: [Equation]
Revised: [Equation]

Original: This approximation is very similar to the true log-likelihood of a Poisson distribution.
Revised: This approximation is very similar to the true log-likelihood of a Poisson distribution.

Original: The key difference is that the log-factorial term over the observations is replaced with the identity over the observations, i.e., log u 1 ,t !
Revised: The key difference is that the log-factorial term over the observations is replaced with the identity over the observations, i.e., log u 1 ,t !

Original: ≈ u 1 ,t .
Revised: ≈ u 1 ,t .

Original: In deep learning applications, the log-factorial is often dropped entirely from the Poisson log-likelihood (see e.g., the Pytorch implementation and default setting 2 ).
Revised: In deep learning applications, the log-factorial is often dropped entirely from the Poisson log-likelihood (see e.g., the Pytorch implementation and default setting 2 ).

Original: It is sometimes replaced with a shifted log-gamma function log k !
Revised: It is sometimes replaced with a shifted log-gamma function log k !

Original: ≈ log Γ( k + 1) or an approximation via Stirling’s formula log k !
Revised: ≈ log Γ( k + 1) or an approximation via Stirling’s formula log k !

Original: ≈ 12 log 2 πk .
Revised: ≈ 12 log 2 πk .

Original: Since Stirling’s formula blows up near zero, and since log-gamma function has a local minimum between zero and one we decided against using these.
Revised: Since Stirling’s formula blows up near zero, and since log-gamma function has a local minimum between zero and one we decided against using these.

Original: However, we observe that this term helps to enforce sparsity in the observations.
Revised: However, we observe that this term helps to enforce sparsity in the observations.

Original: Since this is desirable for inferred spike counts, we replace this term with what is essentially the L 1 norm of u 1 ,t , since by construction u 1 ,t is forced to be positive.
Revised: Since this is desirable for inferred spike counts, we replace this term with what is essentially the L 1 norm of u 1 ,t , since by construction u 1 ,t is forced to be positive.

Original: The remaining terms of the ELBO were not approximated in any special way.
Revised: The remaining terms of the ELBO were not approximated in any special way.

Original: As such we breakdown
Revised: As such we breakdown

Original: L ELBO as follows,
Revised: L ELBO as follows,

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal cost function of CaLFADS becomes
Revised: The ﬁnal cost function of CaLFADS becomes

Original: [Equation]
Revised: [Equation]

Original: where w KL 1 and w KL 2 are KL warmup terms that are typically used in training variational autoencoders.
Revised: where w KL 1 and w KL 2 are KL warmup terms that are typically used in training variational autoencoders.

Original: These terms slowly transition the model from a traditional autoencoder to a variational autoencoder, which helps to prevent pathological behaviour where the KL terms are minimised too early in training.
Revised: These terms slowly transition the model from a traditional autoencoder to a variational autoencoder, which helps to prevent pathological behaviour where the KL terms are minimised too early in training.

Original: This typically results in the model then generating trivial solutions.
Revised: This typically results in the model then generating trivial solutions.

Original: In our case, we use a staggered KL warm-up in which w KL 1 linearly increases from 0 to 1 over epochs 0-100, then w KL 2 linearly increases from 0 to 1 over epochs 100-200.
Revised: In our case, we use a staggered KL warm-up in which w KL 1 linearly increases from 0 to 1 over epochs 0-100, then w KL 2 linearly increases from 0 to 1 over epochs 100-200.

Original: This allows the model to converge to a solution that overﬁts spikes to the ﬂuorescence data ﬁrst, then gradually regularises this proposed set of spikes by ensuring they are well explained by the low-dimensional latent factors.
Revised: This allows the model to converge to a solution that overﬁts spikes to the ﬂuorescence data ﬁrst, then gradually regularises this proposed set of spikes by ensuring they are well explained by the low-dimensional latent factors.

Original: [Figure]
Revised: [Figure]

Original: F
Revised: F

Original: C A LFADS FORWARD
Revised: C A LFADS FORWARD

Original: METHOD PSEUDOCODE
Revised: METHOD PSEUDOCODE

Original: 2 [ h fwdl =0 ,t =1: T , h bwdl =0 ,t =1: T ] = x t =1: T 3 for t ← 1 to T do for l ← 1 to 2 do h fwdl,t =
Revised: 2 [ h fwdl =0 ,t =1: T , h bwdl =0 ,t =1: T ] = x t =1: T 3 for t ← 1 to T do for l ← 1 to 2 do h fwdl,t =

Original: GRU ( h fwd 2 ,t −
Revised: GRU ( h fwd 2 ,t −

Original: [Table]
Revised: [Table]

Original: , [ h fwd
Revised: , [ h fwd

Original: [Table]
Revised: [Table]

Original: ,t , E bwd 1 ,t ])
Revised: ,t , E bwd 1 ,t ])

Original: h
Revised: h

Original: [Table] [Table]
Revised: [Table] [Table]

Original: GRU (
Revised: GRU (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: end 8 end
Revised: end 8 end

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: Dense ([
Revised: Dense ([

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: = exp(
Revised: = exp(

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Dense ([
Revised: Dense ([

Original: [Table]
Revised: [Table]

Original: h bwd 2 ,
Revised: h bwd 2 ,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ∼ N ( µ
Revised: ∼ N ( µ

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Dense (
Revised: Dense (

Original: [Table]
Revised: [Table]

Original: ) 13 for t ← 1 to
Revised: ) 13 for t ← 1 to

Original: [Table]
Revised: [Table]

Original: do c 1 ,t =
Revised: do c 1 ,t =

Original: GRU ( c 1 ,t − 1 , [ h u,fwd 2 ,t , h u,bwd 2 ,t , u 1 ,t − 1 ]) c
Revised: GRU ( c 1 ,t − 1 , [ h u,fwd 2 ,t , h u,bwd 2 ,t , u 1 ,t − 1 ]) c

Original: [Table]
Revised: [Table]

Original: ,t = GRU ( c 2 ,t − 1 , [ h u,fwdl,t , h u,bwd 2 ,t , f t − 1 ]) µ u
Revised: ,t = GRU ( c 2 ,t − 1 , [ h u,fwdl,t , h u,bwd 2 ,t , f t − 1 ]) µ u

Original: [Table]
Revised: [Table]

Original: = Dense ( c 2 ,t ) σ
Revised: = Dense ( c 2 ,t ) σ

Original: [Table]
Revised: [Table]

Original: u
Revised: u

Original: [Table] [Table]
Revised: [Table] [Table]

Original: =
Revised: =

Original: exp( Dense ( c 2 ,t ))
Revised: exp( Dense ( c 2 ,t ))

Original: u 2 ,t ∼ N ( µ u 2 ,t , σ 2 u 2 ,t ) g 2 ,t =
Revised: u 2 ,t ∼ N ( µ u 2 ,t , σ 2 u 2 ,t ) g 2 ,t =

Original: GRU ( g 2 ,t − 1 , u 2 ,t ) f t =
Revised: GRU ( g 2 ,t − 1 , u 2 ,t ) f t =

Original: Dense ( g 2 ,t ) λ t = exp(0 .
Revised: Dense ( g 2 ,t ) λ t = exp(0 .

Original: 5 Dense ( f t ))
Revised: 5 Dense ( f t ))

Original: µ z 1 ,t = Dense ( c 1 ,t ) σ 2 z 1 ,t = exp(0 .
Revised: µ z 1 ,t = Dense ( c 1 ,t ) σ 2 z 1 ,t = exp(0 .

Original: 5 Dense ( c 1 ,t ))
Revised: 5 Dense ( c 1 ,t ))

Original: z 1 ,t ∼ N ( µ z 1 ,t , σ 2 z 1 ,t ) u 1 ,t = ReLU (exp( Dense ([ z 1 ,t , f t ]) − 1))
Revised: z 1 ,t ∼ N ( µ z 1 ,t , σ 2 z 1 ,t ) u 1 ,t = ReLU (exp( Dense ([ z 1 ,t , f t ]) − 1))

Original: g 1 ,t = AR 1( g 1 ,t − 1 , u 1 ,t )
Revised: g 1 ,t = AR 1( g 1 ,t − 1 , u 1 ,t )

Original: ˆ x t =
Revised: ˆ x t =

Original: DyeKinetics ( g 1 ,t ) 28 end
Revised: DyeKinetics ( g 1 ,t ) 28 end

Original: G
Revised: G

Original: C
Revised: C

Original: A LFADS
Revised: A LFADS

Original: LOSS FUNCTION PSEUDOCODE
Revised: LOSS FUNCTION PSEUDOCODE

Original: H S YNTHETIC D ATA G ENERATION
Revised: H S YNTHETIC D ATA G ENERATION

Original: As discussed in the main text, synthetic calcium ﬂuorescence data was generated by embedding a
Revised: As discussed in the main text, synthetic calcium ﬂuorescence data was generated by embedding a

Original: Lorenz Attractor in the dynamics of a spiking neural network.
Revised: Lorenz Attractor in the dynamics of a spiking neural network.

Original: Two models of calcium dynamics were then used to transform spikes to calcium transients, with additive emissions noise added to the resulting traces.
Revised: Two models of calcium dynamics were then used to transform spikes to calcium transients, with additive emissions noise added to the resulting traces.

Original: The Lorenz Attractor is a nonlinear dynamical system with 3 states x 1 , x 2 , x 3 commonly used to study chaotic dynamics.
Revised: The Lorenz Attractor is a nonlinear dynamical system with 3 states x 1 , x 2 , x 3 commonly used to study chaotic dynamics.

Original: The dynamical system is deﬁned by,
Revised: The dynamical system is deﬁned by,

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This system was parameterised in its typical chaotic regime with σ = 10 , β = 8 / 3 , ρ = 28 .
Revised: This system was parameterised in its typical chaotic regime with σ = 10 , β = 8 / 3 , ρ = 28 .

Original: These states were then normalised to have a mean of zero and a range of [ − 1 , 1] .
Revised: These states were then normalised to have a mean of zero and a range of [ − 1 , 1] .

Original: The state of this system x was then randomly projected onto the ﬁring rate of a population of N =neurons,
Revised: The state of this system x was then randomly projected onto the ﬁring rate of a population of N =neurons,

Original: [Equation]
Revised: [Equation]

Original: with W ∼ N (0 , 1 √ N I ) , and b = 1 .
Revised: with W ∼ N (0 , 1 √ N I ) , and b = 1 .

Original: Spike counts in time-bin t for neuron
Revised: Spike counts in time-bin t for neuron

Original: i s i were then sampled using a Poisson distribution
Revised: i s i were then sampled using a Poisson distribution

Original: s i,
Revised: s i,

Original: [Equation]
Revised: [Equation]

Original: 56) where ∆ t is the width of a time-bin.
Revised: 56) where ∆ t is the width of a time-bin.

Original: For our simulations, we used ∆ t = 0 .
Revised: For our simulations, we used ∆ t = 0 .

Original: 1 s .
Revised: 1 s .

Original: Calcium ﬂuorescence traces were then modelled in one of two ways.
Revised: Calcium ﬂuorescence traces were then modelled in one of two ways.

Original: The ﬁrst method modelled calcium concentration in neuron
Revised: The ﬁrst method modelled calcium concentration in neuron

Original: i c i as an exponentially decaying variable with time constant τ = 0 .
Revised: i c i as an exponentially decaying variable with time constant τ = 0 .

Original: 3 and perturbations by spikes
Revised: 3 and perturbations by spikes

Original: [Equation]
Revised: [Equation]

Original: Fluorescence was then modelled by adding white emissions noise with standard deviation σ F = 0 .
Revised: Fluorescence was then modelled by adding white emissions noise with standard deviation σ F = 0 .

Original: [Equation]
Revised: [Equation]

Original: The second method for generating synthetic calcium traces added an intermediary step between calcium inﬂux and observed ﬂuorescence.
Revised: The second method for generating synthetic calcium traces added an intermediary step between calcium inﬂux and observed ﬂuorescence.

Original: As previously, spikes were integrated with a slow-varying exponentially decaying variable c as described in Equation 57.
Revised: As previously, spikes were integrated with a slow-varying exponentially decaying variable c as described in Equation 57.

Original: A hill function was used to capture the nonlinear binding kinetics of calcium to the indicator dye,
Revised: A hill function was used to capture the nonlinear binding kinetics of calcium to the indicator dye,

Original: [Equation]
Revised: [Equation]

Original: 59) with hill coefﬁcient n = 2 , and γ = 0 . 0001 .
Revised: 59) with hill coefﬁcient n = 2 , and γ = 0 . 0001 .

Original: These parameters were chosen based on parameter ﬁts in Deneux et al.
Revised: These parameters were chosen based on parameter ﬁts in Deneux et al.

Original: Finally white noise was added to these resulting traces to provide synthetic ﬂuorescence traces,
Revised: Finally white noise was added to these resulting traces to provide synthetic ﬂuorescence traces,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: I C A LFADS EXAMPLE OUTPUTS AND SCATTER PLOTS ON SYNTHETIC BENCHMARK DATASETS
Revised: I C A LFADS EXAMPLE OUTPUTS AND SCATTER PLOTS ON SYNTHETIC BENCHMARK DATASETS

Original: [Figure]
Revised: [Figure]

Original: Figure S2 shows examples of the performance of CaLFADS in reconstructing the ﬂuorescence traces (Fig S2A), spike rates (Fig S2B), spike counts (Fig S2C) and Lorenz attractor states (Fig S2D) when using a linear model of synthetic calcium dynamics.
Revised: Figure S2 shows examples of the performance of CaLFADS in reconstructing the ﬂuorescence traces (Fig S2A), spike rates (Fig S2B), spike counts (Fig S2C) and Lorenz attractor states (Fig S2D) when using a linear model of synthetic calcium dynamics.

Original: Visually, it can be seen that the model achieves a very close ﬁt to the ﬂuorescence traces, spike rates, and Lorenz dynamics.
Revised: Visually, it can be seen that the model achieves a very close ﬁt to the ﬂuorescence traces, spike rates, and Lorenz dynamics.

Original: The model also captures spike-timing, with these spike trains appearing smoothed.
Revised: The model also captures spike-timing, with these spike trains appearing smoothed.

Original: For comparison, Figure S3 show examples of OASIS+LFADS reconstructions.
Revised: For comparison, Figure S3 show examples of OASIS+LFADS reconstructions.

Original: A key difference is that the spike counts are not smooth as spike counts are discretised from the deconvolved transient amplitudes, and there is greater variance in reconstruction quality at high spike rates as shown by the scatter plot in Figure S3F. Nevertheless, OASIS+LFADS also provides a good ﬁt to ﬂuorescence traces, spike rates, spike counts and lorenz dynamics.
Revised: A key difference is that the spike counts are not smooth as spike counts are discretised from the deconvolved transient amplitudes, and there is greater variance in reconstruction quality at high spike rates as shown by the scatter plot in Figure S3F. Nevertheless, OASIS+LFADS also provides a good ﬁt to ﬂuorescence traces, spike rates, spike counts and lorenz dynamics.

Original: I. L ORENZ A TTRACTOR : N ONLINEAR C ALCIUM M ODEL
Revised: I. L ORENZ A TTRACTOR : N ONLINEAR C ALCIUM M ODEL

Original: Figure S4 shows examples of the performance of CaLFADS in reconstructing the ﬂuorescence traces (Fig S4A), spike rates (Fig S4B), spike counts (Fig S4C) and Lorenz attractor states (Fig S4D) when using a nonlinear model to generated synthetic calcium traces.
Revised: Figure S4 shows examples of the performance of CaLFADS in reconstructing the ﬂuorescence traces (Fig S4A), spike rates (Fig S4B), spike counts (Fig S4C) and Lorenz attractor states (Fig S4D) when using a nonlinear model to generated synthetic calcium traces.

Original: As with the linear calcium model, it can be seen that the model achieves a close reconstruction of to the Lorenz dynamics.
Revised: As with the linear calcium model, it can be seen that the model achieves a close reconstruction of to the Lorenz dynamics.

Original: However, the reconstructions of other network variables have deteriorated to a much geater extent.
Revised: However, the reconstructions of other network variables have deteriorated to a much greater extent.

Original: A similar trend can be seen in Figure S5 for OASIS+LFADS.
Revised: A similar trend can be seen in Figure S5 for OASIS+LFADS.

Original: J M OUSE V IS P DATASET , DECODING NETWORK AND EXAMPLE FACTORSJ. M OUSE V IS P DATASET
Revised: J M OUSE V IS P DATASET , DECODING NETWORK AND EXAMPLE FACTORS

Revised: J. M OUSE V IS P DATASET

Original: The 2-photon calcium imaging data used here is part of the OpenScope project dataset (Anonymous, 2018), collected by the Allen Institute for Brain Science (AIBS) in Seattle, WA.
Revised: The 2-photon calcium imaging data used here is part of the OpenScope project dataset (Anonymous, 2018), collected by the Allen Institute for Brain Science (AIBS) in Seattle, WA.

Original: All animal procedures were approved by the Institutional Animal Care and Use Committee at the AIBS.
Revised: All animal procedures were approved by the Institutional Animal Care and Use Committee at the AIBS.

Original: Neuronal activity was recorded in head-ﬁxed, awake Cux2-Cre mice on a running wheel (de Vries et al., 2018) (Fig 3A) expressing the calcium indicator GCamP6f
Revised: Neuronal activity was recorded in head-ﬁxed, awake Cux2-Cre mice on a running wheel (de Vries et al., 2020) (Fig 3A) expressing the calcium indicator GCamP6f

Original: (Chen et al., 2013) in layer 2/3 pyramidal cells of VisP (Fig 3B).
Revised: (Chen et al., 2013) in layer 2/3 pyramidal cells of VisP (Fig 3B).

Original: The mice were ﬁrst habituated to a repeating, expected stimulus over 6 days, after which unexpected trials were introduced.
Revised: The mice were ﬁrst habituated to a repeating, expected stimulus over 6 days, after which unexpected trials were introduced.

Original: The stimuli were adapted from (Homann et al., 2017).
Revised: The stimuli were adapted from (Homann et al., 2017).

Original: In each expected trial, 4 consecutive sets of 30 Gabor patches appeared in sequence for 300 ms each, followed by 300 ms of grey screen (A, B, C, D, grey).
Revised: In each expected trial, 4 consecutive sets of 30 Gabor patches appeared in sequence for 300 ms each, followed by 300 ms of grey screen (A, B, C, D, grey).

Original: For each set, the locations and sizes of the
Revised: For each set, the locations and sizes of the

Original: Gabor patches were held constant within a session.
Revised: Gabor patches were held constant within a session.

Original: However, within each trial, each Gabor patch’s orientation was sampled from a von Mises distribution, with trial mean sampled from { 0, 45, 90, 135 } ◦ and standard deviation 0.25 ◦ .
Revised: However, within each trial, each Gabor patch’s orientation was sampled from a von Mises distribution, with trial mean sampled from { 0, 45, 90, 135 } ◦ and standard deviation 0.25 ◦ .

Original: In expected trials, occurring 10% of the time, the D set was replaced with a distinct set E, with its own locations and sizes.
Revised: In expected trials, occurring 10% of the time, the D set was replaced with a distinct set E, with its own locations and sizes.

Original: In addition, the Gabor patch orientations for E sets were sampled from a von Mises distribution with mean shifted positively by 90 ◦ (Fig 3C).
Revised: In addition, the Gabor patch orientations for E sets were sampled from a von Mises distribution with mean shifted positively by 90 ◦ (Fig 3C).

Original: Processed ﬂuorescence (dF/F) traces were extracted from the calcium imaging recordings for the putative neurons identiﬁed, as described in (de Vries et al., 2018).
Revised: Processed ﬂuorescence (dF/F) traces were extracted from the calcium imaging recordings for the putative neurons identiﬁed, as described in (de Vries et al., 2020). [Figure]

Original: J.
Revised: Figures S6 and S7 show samples of actual ﬂuorescence traces (A), the corresponding inferred spike count traces (B), and inferred latent factors (C) generated as the output of the CaLFADS and OASIS+LFADS models, respectively. Figure S6A additionally shows examples of reconstructed and actual ﬂuorescence traces (A, left column). As can be seen in the ﬁgures, the inferred latent factors of CaLFADS are smoother and less noisy compared to those of the OASIS+LFADS. This is important as the quality of the inferred factors is crucial for decoding purposes, such as the classiﬁcation task described in Appendix J.3.

Original: [Figure]
Revised: [Figure]

Original: EXAMPLE
Revised: EXAMPLE

Original: [Figure]
Revised: [Figure]

Original: Figure S6 shows a schematic of the non-linear model used to decode from the latent factors whether a trial contained an expected or unexpected frame (decoding performances reported in Fig 3D).
Revised: Figure S7 shows a schematic of the non-linear model used to decode from the latent factors whether a trial contained an expected or unexpected frame (decoding performances reported in Fig 3D).

Original: In short, the non-linear decoder comprised of a single-layer GRU, followed by a linear layer and a sigmoid decision function.
Revised: In short, the non-linear decoder comprised of a single-layer GRU, followed by a linear layer and a sigmoid decision function.

Original: We show an example set of latent factors (Figure S6 left) inferred in expected (blue) and unexpected (red) trials.
Revised: We show an example set of latent factors (Figure S7 left) inferred in expected (blue) and unexpected (red) trials.

Original: From these, one can see that CaLFADS was able to infer smooth latent factors from the calcium imaging data that did not show signs of over-ﬁtting to noise.
Revised: From these, one can see that CaLFADS was able to infer smooth latent factors from the calcium imaging data that did not show signs of over-ﬁtting to noise.