Original: L E M O NAD E : L EARNED M OTIF AND N EURONAL A SSEMBLY D ETECTION IN CALCIUM IMAGING VIDEOS
Revised: L E M O NAD E : L EARNED M OTIF AND N EURONAL A SSEMBLY D ETECTION IN CALCIUM IMAGING VIDEOS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neuronal assemblies, loosely deﬁned as subsets of neurons with reoccurring spatiotemporally coordinated activation patterns, or "motifs", are thought to be building blocks of neural representations and information processing.
Revised: Neuronal assemblies, loosely deﬁned as subsets of neurons with reoccurring spatiotemporally coordinated activation patterns, or "motifs", are thought to be building blocks of neural representations and information processing.

Original: We here propose
Revised: We here propose

Original: LeMoNADe, a new exploratory data analysis method that facilitates hunting for motifs in calcium imaging videos, the dominant microscopic functional imaging modality in neurophysiology.
Revised: LeMoNADe, a new exploratory data analysis method that facilitates hunting for motifs in calcium imaging videos, the dominant microscopic functional imaging modality in neurophysiology.

Original: Our nonparametric method extracts motifs directly from videos, bypassing the difﬁcult intermediate step of spike extraction.
Revised: Our nonparametric method extracts motifs directly from videos, bypassing the difﬁcult intermediate step of spike extraction.

Original: Our technique augments variational autoencoders with a discrete stochastic node and we show in detail how a differentiable reparametrization and relaxation can be used.
Revised: Our technique augments variational autoencoders with a discrete stochastic node, and we show in detail how a differentiable reparametrization and relaxation can be used.

Original: An evaluation on simulated data, with available ground truth, reveals excellent quantitative performance.
Revised: An evaluation on simulated data, with available ground truth, reveals excellent quantitative performance.

Original: In real video data acquired from brain slices, with no ground truth available, LeMoNADe uncovers nontrivial candidate motifs that can help generate hypotheses for more focused biological investigations.
Revised: In real video data acquired from brain slices, with no ground truth available, LeMoNADe uncovers nontrivial candidate motifs that can help generate hypotheses for more focused biological investigations.

Original: I NTRODUCTION
Revised: I NTRODUCTION

Original: Seventy years after being postulated by Hebb (1949), the existence and importance of reoccurring spatio-temporally coordinated neuronal activation patterns (motifs), also known as neuronal assemblies, is still ﬁercely debated (Marr et al., 1991; Singer, 1993; Nicolelis et al., 1997; Ikegaya et al., 2004; Cossart & Sansonetti, 2004; Buzsáki, 2004; Mokeichev et al., 2007; Pastalkova et al., 2008;
Revised: Seventy years after being postulated by Hebb (1949), the existence and importance of reoccurring spatio-temporally coordinated neuronal activation patterns (motifs), also known as neuronal assemblies, is still ﬁercely debated (Marr et al., 1991; Singer, 1993; Nicolelis et al., 1997; Ikegaya et al., 2004; Cossart & Sansonetti, 2004; Buzsáki, 2004; Mokeichev et al., 2007; Pastalkova et al., 2008;

Original: Stevenson & Kording, 2011; Ahrens et al., 2013; Carrillo-Reid et al., 2015).
Revised: Stevenson & Kording, 2011; Ahrens et al., 2013; Carrillo-Reid et al., 2015).

Original: Calcium imaging, a microscopic video technique that enables the concurrent observation of hundreds of neurons in vitro and in vivo (Denk et al., 1990; Helmchen & Denk, 2005; Flusberg et al., 2008), is best suited to witness such motifs if they indeed exist. 
Revised: Calcium imaging, a microscopic video technique that enables the concurrent observation of hundreds of neurons in vitro and in vivo (Denk et al., 1990; Helmchen & Denk, 2005; Flusberg et al., 2008), is best suited to witness such motifs if they indeed exist. [Figure] Identiﬁcation e.g. Pnev- matikakis et al. ; Zhou et al.

Revised: Spike Time Extraction e.g. Speiser et al.

Revised: Motif Detection e.g. Russo & Durstewitz (2017); Peter et al.

Original:  In recent years, a variety of methods have been developed to identify neuronal assemblies.
Revised: NeuronalAssemblies Motif Detection in Ca Videos LeMoNADe In recent years, a variety of methods have been developed to identify neuronal assemblies.

Original: These methods range from approaches for the detection of synchronous spiking, up to more advanced methods for the detection of arbitrary spatio-temporal ﬁring patterns (Comon, 1994;
Revised: These methods range from approaches for the detection of synchronous spiking, up to more advanced methods for the detection of arbitrary spatio-temporal ﬁring patterns (Comon, 1994;

Original: Nicolelis et al., 1995;
Revised: Nicolelis et al., 1995;

Original: Grün et al., 2002a;b; Lopes-dos Santos et al., 2013; Russo & Durstewitz, 2017; Peter et al., 2017).
Revised: Grün et al., 2002a;b; Lopes-dos Santos et al., 2013; Russo & Durstewitz, 2017; Peter et al., 2017).

Original: All of these methods, however, require a spike time matrix as input.
Revised: All of these methods, however, require a spike time matrix as input.

Original: Generating such a spike time matrix from calcium imaging data requires the extraction of individual cells and discrete spike times.
Revised: Generating such a spike time matrix from calcium imaging data requires the extraction of individual cells and discrete spike times.

Original: Again, many methods have been proposed for these tasks (Mukamel et al., 2009;
Revised: Again, many methods have been proposed for these tasks (Mukamel et al., 2009;

Original: Pnevmatikakis & Paninski, 2013; Pnevmatikakis et al., 2013; Diego et al., 2013; Diego & Hamprecht, 2013; Pachitariu et al., 2013; Pnevmatikakis et al., 2014; Diego & Hamprecht, 2014; Kaifosh et al., 2014; Pnevmatikakis et al., 2016; Apthorpe et al., 2016; Inan et al., 2017; Spaen et al., 2017;
Revised: Pnevmatikakis & Paninski, 2013; Pnevmatikakis et al., 2013; Diego et al., 2013; Diego & Hamprecht, 2013; Pachitariu et al., 2013; Pnevmatikakis et al., 2014; Diego & Hamprecht, 2014; Kaifosh et al., 2014; Pnevmatikakis et al., 2016; Apthorpe et al., 2016; Inan et al., 2017; Spaen et al., 2017;

Original: Klibisz et al., 2017; Speiser et al., 2017; Zhou et al., 2018).
Revised: Klibisz et al., 2017; Speiser et al., 2017; Zhou et al., 2018).

Original: Given the low signal-to-noise ratios (SNR), large background ﬂuctuations, non-linearities, and strong temporal smoothing due to the calcium dynamics itself as well as that of calcium indicators, it is impressive how well some of these methods perform, thanks to modern recording technologies and state-of-the-art regularization and inference (Pnevmatikakis et al., 2016; Zhou et al., 2018).
Revised: Given the low signal-to-noise ratios (SNR), large background ﬂuctuations, non-linearities, and strong temporal smoothing due to the calcium dynamics itself as well as that of calcium indicators, it is impressive how well some of these methods perform, thanks to modern recording technologies and state-of-the-art regularization and inference (Pnevmatikakis et al., 2016; Zhou et al., 2018).

Original: Still, given the difﬁculty of this data, errors in segmentation and spike extraction are unavoidable, and adversely affect downstream processing steps that do not have access to the raw data.
Revised: Still, given the difﬁculty of this data, errors in segmentation and spike extraction are unavoidable, and adversely affect downstream processing steps that do not have access to the raw data.

Original: Hence, properly annotating data and correcting the output from automatic segmentation can still take up a huge amount of time.
Revised: Hence, properly annotating data and correcting the output from automatic segmentation can still take up a huge amount of time.

Original: In this paper, we propose LeMoNADe (Learned Motif and Neuronal Assembly Detection) , a variational autoencoder (VAE) based framework speciﬁcally designed to identify repeating ﬁring motifs with arbitrary temporal structure directly in calcium imaging data (see ﬁgure 1).The encoding and [Figure] Cell Identiﬁcation e.g. Pnev- matikakis et al.
Revised: In this paper, we propose LeMoNADe (Learned Motif and Neuronal Assembly Detection) , a variational autoencoder (VAE) based framework speciﬁcally designed to identify repeating ﬁring motifs with arbitrary temporal structure directly in calcium imaging data (see ﬁgure 1).

Original: ; Zhou et al.

Original: Spike Time Extraction e.g. Speiser et al.

Original: Motif Detection e.g. Russo & Durstewitz (2017); Peter et al.

Original: NeuronalAssemblies Motif Detection in Ca Videos

Original: LeMoNADe decoding networks are set up such that motifs can be extracted directly from the decoding ﬁlters, and their activation times from the latent space (see sec. 3).
Revised: The encoding and decoding networks are set up such that motifs can be extracted directly from the decoding ﬁlters, and their activation times from the latent space (see sec. 3).

Original: Motivated by the sparse nature of neuronal activity we replace the Gaussian priors used in standard VAE.
Revised: Motivated by the sparse nature of neuronal activity we replace the Gaussian priors used in standard VAE.

Original: Instead we place Bernoulli priors on the latent variables to yield sparse and sharply peaked motif activations (sec. 3.1).
Revised: Instead we place Bernoulli priors on the latent variables to yield sparse and sharply peaked motif activations (sec. 3.1).

Original: The choice of discrete Bernoulli distributions makes it necessary to use a BinConcrete relaxation and the Gumbel-softmax reparametrization trick (Maddison et al., 2016; Jang et al., 2017) to enable gradient descent techniques with low variance (sec. 3.3).
Revised: The choice of discrete Bernoulli distributions makes it necessary to use a BinConcrete relaxation and the Gumbel-softmax reparametrization trick (Maddison et al., 2016; Jang et al., 2017) to enable gradient descent techniques with low variance (sec. 3.3).

Original: We add a β -coefﬁcient (Higgins et al., 2017) to the loss function in order to adapt the regularization to the properties of the data (sec. 3.3).
Revised: We add a β -coefﬁcient (Higgins et al., 2017) to the loss function in order to adapt the regularization to the properties of the data (sec. 3.3).

Original: Furthermore, we propose a training scheme which allows us to process videos of arbitrary length in a computationally efﬁcient way (sec. 3.4).
Revised: Furthermore, we propose a training scheme which allows us to process videos of arbitrary length in a computationally efﬁcient way (sec. 3.4).

Original: On synthetically generated datasets the proposed method performs as well as a state-of-the-art motif detection method that requires the extraction of individual cells (sec. 4.1).
Revised: On synthetically generated datasets the proposed method performs as well as a state-of-the-art motif detection method that requires the extraction of individual cells (sec. 4.1).

Original: Finally, we detect possible repeating motifs in two ﬂuorescent microscopy datasets from hippocampal slice cultures (sec. 4.2).
Revised: Finally, we detect possible repeating motifs in two ﬂuorescent microscopy datasets from hippocampal slice cultures (sec. 4.2).

Original: A PyTorch implementation of the proposed method is released on GitHub 1 .
Revised: A PyTorch implementation of the proposed method is released at https://github.com/EKirschbaum/LeMoNADe .

Original:  R ELATED 
Revised:  R ELATED W ORK Autoencoder and variational autoencoder

Original: W ORK Autoencoder and variational autoencoder

Original: Variational Autoencoders (VAEs) were introduced by Kingma & Welling (2014) and have become a popular method for unsupervised generative deep learning.
Revised: Variational Autoencoders (VAEs) were introduced by Kingma & Welling (2014) and have become a popular method for unsupervised generative deep learning.

Original: They consist of an encoder, mapping a data point into a latent representation, and a decoder whose task is to restore the original data and to generate samples from this latent space.
Revised: They consist of an encoder, mapping a data point into a latent representation, and a decoder whose task is to restore the original data and to generate samples from this latent space.

Original: However, the original VAE lacks an interpretable latent space.
Revised: However, the original VAE lacks an interpretable latent space.

Original: Recent suggestions on solving this problem have been modiﬁcations of the loss term (Higgins et al., 2017) or a more structured latent space (Johnson et al., 2016; Deng et al., 2017).
Revised: Recent suggestions on solving this problem have been modiﬁcations of the loss term (Higgins et al., 2017) or a more structured latent space (Johnson et al., 2016; Deng et al., 2017).

Original: VAE have also been successfully used on video sequences.
Revised: VAE have also been successfully used on video sequences.

Original: Li & Mandt (2018) learn a disentangled representation to manipulate content in cartoon video clips, while Goyal et al.
Revised: Li & Mandt (2018) learn a disentangled representation to manipulate content in cartoon video clips, while Goyal et al.

Original: (2017) combine
Revised: (2017) combine

Original: VAEs with nested Chinese Restaurant Processes to learn a hierarchical representation of video data.
Revised: VAEs with nested Chinese Restaurant Processes to learn a hierarchical representation of video data.

Original: Johnson et al.
Revised: Johnson et al.

Original: (2016) use a latent switching linear dynamical system (SLDS) model combined with a structured variational autoencoder to segment and categorize mouse behavior from raw depth videos.
Revised: (2016) use a latent switching linear dynamical system (SLDS) model combined with a structured variational autoencoder to segment and categorize mouse behavior from raw depth videos.

Original: Unfortunately, this model is not directly applicable to the task of identifying motifs with temporal structure from calcium imaging data for the following reasons: Firstly, neuronal assemblies are expected to extend over multiple frames.
Revised: Unfortunately, this model is not directly applicable to the task of identifying motifs with temporal structure from calcium imaging data for the following reasons: Firstly, neuronal assemblies are expected to extend over multiple frames.

Original: Since in the model by Johnson et al.
Revised: Since in the model by Johnson et al.

Original: the underlying latent process is a relatively simple ﬁrst-order Markovian (switching) linear process, representing longer-term temporal dependencies will be very hard to achieve due to the usually exponential forgetting in such systems.
Revised: the underlying latent process is a relatively simple ﬁrst-order Markovian (switching) linear process, representing longer-term temporal dependencies will be very hard to achieve due to the usually exponential forgetting in such systems.

Original: Secondly, in the model of Johnson et al.
Revised: Secondly, in the model of Johnson et al.

Original: each frame is generated from exactly one of M latent states.
Revised: each frame is generated from exactly one of M latent states.

Original: For calcium imaging, however, most frames are not generated by one of the M motifs but from noise, and different motifs could also temporally overlap which is also not possible in the model by Johnson et al. (2016).
Revised: For calcium imaging, however, most frames are not generated by one of the M motifs but from noise, and different motifs could also temporally overlap which is also not possible in the model by Johnson et al. (2016).

Original: Closest to our goal of detecting motifs in video data is the work described in Bascol et al.
Revised: Closest to our goal of detecting motifs in video data is the work described in Bascol et al.

Original: In this approach, a convolutional autoencoder is combined with a number of functions and regularization terms to enforce interpretability both in the convolutional ﬁlters and the latent space.
Revised: In this approach, a convolutional autoencoder is combined with a number of functions and regularization terms to enforce interpretability both in the convolutional ﬁlters and the latent space.

Original: This method was successfully used to detect patterns in data with document structure, including optical ﬂow features of videos.
Revised: This method was successfully used to detect patterns in data with document structure, including optical ﬂow features of videos.

Original: However, as the cells observed in calcium imaging are spatially stationary and have varying luminosity, the extraction of optical ﬂow features makes no sense.
Revised: However, as the cells observed in calcium imaging are spatially stationary and have varying luminosity, the extraction of optical ﬂow features makes no sense.

Original: Hence this method is not applicable to the task of detecting neuronal assemblies in calcium imaging data.
Revised: Hence this method is not applicable to the task of detecting neuronal assemblies in calcium imaging data.

Original: Cell segmentation and spike time extraction from calcium imaging data
Revised: Cell segmentation and spike time extraction from calcium imaging data

Original: Various methods have been proposed for automated segmentation and signal extraction from calcium imaging data.
Revised: Various methods have been proposed for automated segmentation and signal extraction from calcium imaging data.

Original: Most of them are based on non-negative matrix factorization (Mukamel et al., 2009; Pnevmatikakis & Paninski, 2013; Pnevmatikakis et al., 2013; 2014; Diego & Hamprecht, 2014; Pnevmatikakis et al., 2016; Inan et al., 2017; Zhou et al., 2018), clustering (Kaifosh et al., 2014; Spaen et al., 2017), and dictionary learning (Diego et al., 2013; Diego & Hamprecht, 2013;
Revised: Most of them are based on non-negative matrix factorization (Mukamel et al., 2009; Pnevmatikakis & Paninski, 2013; Pnevmatikakis et al., 2013; 2014; Diego & Hamprecht, 2014; Pnevmatikakis et al., 2016; Inan et al., 2017; Zhou et al., 2018), clustering (Kaifosh et al., 2014; Spaen et al., 2017), and dictionary learning (Diego et al., 2013; Diego & Hamprecht, 2013;

Original: Pachitariu et al., 2013).
Revised: Pachitariu et al., 2013).

Original: Recent approaches started to use deep learning for the analysis of calcium imaging data.
Revised: Recent approaches started to use deep learning for the analysis of calcium imaging data.

Original: Apthorpe et al.
Revised: Apthorpe et al.

Original: (2016) and Klibisz et al.
Revised: (2016) and Klibisz et al.

Original: (2017) use convolutional neural networks (CNNs) to identify neuron locations and Speiser et al.
Revised: (2017) use convolutional neural networks (CNNs) to identify neuron locations and Speiser et al.

Original: (2017) use a VAE combined with different models for calcium dynamics to extract spike times from the calcium transients.
Revised: (2017) use a VAE combined with different models for calcium dynamics to extract spike times from the calcium transients.

Original: Although many sophisticated methods have been proposed, the extraction of cells and spike times from calcium imaging data can still be prohibitively laborious and require manual annotation and correction, with the accuracy of these methods being limited by the quality of the calcium recordings.
Revised: Although many sophisticated methods have been proposed, the extraction of cells and spike times from calcium imaging data can still be prohibitively laborious and require manual annotation and correction, with the accuracy of these methods being limited by the quality of the calcium recordings.

Original: Furthermore, some of the mentioned methods are specially designed for two-photon microscopy, whereas only few methods are capable to deal with the low SNR and large background ﬂuctuations in single-photon and microendoscopic imaging (Flusberg et al., 2008; Ghosh et al., 2011).
Revised: Furthermore, some of the mentioned methods are specially designed for two-photon microscopy, whereas only few methods are capable to deal with the low SNR and large background ﬂuctuations in single-photon and microendoscopic imaging (Flusberg et al., 2008; Ghosh et al., 2011).

Original: Additional challenges for these methods are factors such as non-Gaussian noise, non-cell background activity and seemingly overlapping cells which are out of focus (Inan et al., 2017).
Revised: Additional challenges for these methods are factors such as non-Gaussian noise, non-cell background activity and seemingly overlapping cells which are out of focus (Inan et al., 2017).

Original: Neuronal assembly detection
Revised: Neuronal assembly detection

Original: The identiﬁcation of neuronal assemblies in spike time matrices has been studied from different perspectives.
Revised: The identiﬁcation of neuronal assemblies in spike time matrices has been studied from different perspectives.

Original: For the detection of joint (strictly synchronous) spike events across multiple neurons, rather simple methods based on PCA or ICA have been proposed (Comon, 1994; Nicolelis et al., 1995; Lopes-dos Santos et al., 2013), as well as more sophisticated statistical methods such as unitary event analysis (Grün et al., 2002a;b).
Revised: For the detection of joint (strictly synchronous) spike events across multiple neurons, rather simple methods based on PCA or ICA have been proposed (Comon, 1994; Nicolelis et al., 1995; Lopes-dos Santos et al., 2013), as well as more sophisticated statistical methods such as unitary event analysis (Grün et al., 2002a;b).

Original: Higher-order correlations among neurons and sequential spiking motifs such as synﬁre chains can be identiﬁed using more advanced statistical tests (Staude et al., 2010a;b; Gerstein et al., 2012).
Revised: Higher-order correlations among neurons and sequential spiking motifs such as synﬁre chains can be identiﬁed using more advanced statistical tests (Staude et al., 2010a;b; Gerstein et al., 2012).

Original: The identiﬁcation of cell assemblies with arbitrary spatio-temporal structure has been addressed only quite recently.
Revised: The identiﬁcation of cell assemblies with arbitrary spatio-temporal structure has been addressed only quite recently.

Original: One approach recursively merges sets of units into larger groups based on their joint spike count probabilities evaluated across multiple different time lags (Russo & Durstewitz, 2017).
Revised: One approach recursively merges sets of units into larger groups based on their joint spike count probabilities evaluated across multiple different time lags (Russo & Durstewitz, 2017).

Original: Another method uses sparse convolutional coding (SCC) for reconstructing the spike matrix as a convolution of spatio-temporal motifs and their activations in time (Peter et al., 2017).
Revised: Another method uses sparse convolutional coding (SCC) for reconstructing the spike matrix as a convolution of spatio-temporal motifs and their activations in time (Peter et al., 2017).

Original: An extension of this method uses a group sparsity regularization to identify the correct number of motifs (Mackevicius et al., 2018). 
Revised: An extension of this method uses a group sparsity regularization to identify the correct number of motifs (Mackevicius et al., 2018). [Figure]

Original: To the authors’ knowledge, solely Diego & Hamprecht (2013) address the detection of neuronal assemblies directly from calcium imaging data.
Revised: To the authors’ knowledge, solely Diego & Hamprecht (2013) address the detection of neuronal assemblies directly from calcium imaging data.

Original: This method, however, only aims at identifying synchronously ﬁring neurons, whereas the method proposed in this paper can identify also assemblies with more complex temporal ﬁring patterns.
Revised: This method, however, only aims at identifying synchronously ﬁring neurons, whereas the method proposed in this paper can identify also assemblies with more complex temporal ﬁring patterns.

Original:  M ETHOD
Revised:  M ETHOD

Original: LeMoNADe is a VAE based latent variable method, speciﬁcally designed for the unsupervised detection of repeating motifs with temporal structure in video data.
Revised: LeMoNADe is a VAE based latent variable method, speciﬁcally designed for the unsupervised detection of repeating motifs with temporal structure in video data.

Original: The data x is reconstructed as a convolution of motifs and their activation time points as displayed in ﬁgure 2a.
Revised: The data x is reconstructed as a convolution of motifs and their activation time points as displayed in ﬁgure 2a.

Original: The VAE is set up such that the latent variables z contain the activations of the motifs, while the decoder encapsulates the ﬁring motifs of the cells as indicated in ﬁgure 2b.
Revised: The VAE is set up such that the latent variables z contain the activations of the motifs, while the decoder encapsulates the ﬁring motifs of the cells as indicated in ﬁgure 2b.

Original: The proposed generative model is displayed in ﬁgure 3.
Revised: The proposed generative model is displayed in ﬁgure 3.

Original: The great beneﬁt of this generative model in combination with the proposed VAE is the possibility to directly extract the temporal motifs and their activations and at the same time take into account the sparse nature of neuronal assemblies.
Revised: The great beneﬁt of this generative model in combination with the proposed VAE is the possibility to directly extract the temporal motifs and their activations and at the same time take into account the sparse nature of neuronal assemblies.

Original: 3. T HE L E M O NAD E MODEL
Revised: 3. T HE L E M O NAD E MODEL

Original: In the proposed model the dataset consists of a single video x ∈ R T × P × P (cid:48) with T frames of P × P (cid:48) pixels each.
Revised: In the proposed model the dataset consists of a single video x ∈ R T × P × P (cid:48) with T frames of P × P (cid:48) pixels each.

Original: We assume this video to be an additive mixture of M repeating motifs of maximum[Figure] temporal length F .
Revised: We assume this video to be an additive mixture of M repeating motifs of maximum temporal length F .

Original: At each time frame t = 1 , . . .
Revised: At each time frame t = 1 , . . .

Original: , T , and for each motif m = 1 , . .
Revised: , T , and for each motif m = 1 , . .

Original: , M , a latent random variable z mt ∈ { 0 , 1 } is drawn from a prior distribution p a ( z ) .
Revised: , M , a latent random variable z mt ∈ { 0 , 1 } is drawn from a prior distribution p a ( z ) .

Original: The variable z mt indicates  whether motif m is activated in frame t or not.
Revised: The variable z mt indicates [Figure] whether motif m is activated in frame t or not.

Original: The video x is then generated from the conditional distribution p θ ( x | z ) with parameters θ .
Revised: The video x is then generated from the conditional distribution p θ ( x | z ) with parameters θ .

Original: In order to infer the latent activations z the posterior p θ ( z | x ) is needed.
Revised: In order to infer the latent activations z the posterior p θ ( z | x ) is needed.

Original: However, the true posterior p θ ( z | x ) is intractable, but it can be approximated by introducing the recognition model (or approximate posterior) q φ ( z | x ) .
Revised: However, the true posterior p θ ( z | x ) is intractable, but it can be approximated by introducing the recognition model (or approximate posterior) q φ ( z | x ) .

Original: We assume that the recognition model q φ ( z | x ) factorizes into the M motifs and T time steps of the video.
Revised: We assume that the recognition model q φ ( z | x ) factorizes into the M motifs and T time steps of the video.

Original: In contrast to most VAE, we further assume that each latent variable z mt is Bernoulli-distributed with parameter α mt ( x ; φ )
Revised: In contrast to most VAE, we further assume that each latent variable z mt is Bernoulli-distributed with parameter α mt ( x ; φ )

Original: [Equation]
Revised: [Equation]

Original: We sample the activations z in the latent space from the Bernoulli distributions to enforce sparse, sharply peaked activations.
Revised: We sample the activations z in the latent space from the Bernoulli distributions to enforce sparse, sharply peaked activations.

Original: The parameters α mt ( x ; φ ) are given by a CNN with parameters φ .
Revised: The parameters α mt ( x ; φ ) are given by a CNN with parameters φ .

Original: The corresponding plate diagram and proposed generative and recognition model are shown in ﬁgure 3.
Revised: The corresponding plate diagram and proposed generative and recognition model are shown in ﬁgure 3.

Original: 3. T HE VAE OBJECTIVE
Revised: 3. T HE VAE OBJECTIVE

Original: In order to learn the variational parameters, the KL-divergence between approximate and true posterior KL ( q φ ( z | x ) (cid:107) p θ ( z | x )) is minimized.
Revised: In order to learn the variational parameters, the KL-divergence between approximate and true posterior KL ( q φ ( z | x ) (cid:107) p θ ( z | x )) is minimized.

Original: Instead of minimizing this KL-divergence, we can[Figure] also maximize the variational lower bound L ( θ, φ ; x ) (ELBO) (see e.g. Blei et al.
Revised: Instead of minimizing this KL-divergence, we can also maximize the variational lower bound L ( θ, φ ; x ) (ELBO) (see e.g. Blei et al.

Original: [Equation]
Revised: [Equation]

Original: In order to optimize the ELBO, the gradients w.r.t.
Revised: In order to optimize the ELBO, the gradients w.r.t.

Original: the variational parameters φ and the generative parameters θ have to be computed.
Revised: the variational parameters φ and the generative parameters θ have to be computed.

Original: The gradient w.r.t.
Revised: The gradient w.r.t.

Original: φ , however, cannot be computed easily, since the expectation in eq.
Revised: φ , however, cannot be computed easily, since the expectation in eq.

Original: (2) depends on φ .
Revised: (2) depends on φ .

Original: A reparameterization trick (Kingma et al., 2015) is used to overcome this problem: the random variable z ∼ q φ ( z | x ) is reparameterized using a differentiable transformation h φ ( ε, x ) of a noise variable ε such that
Revised: A reparameterization trick (Kingma et al., 2015) is used to overcome this problem: the random variable z ∼ q φ ( z | x ) is reparameterized using a differentiable transformation h φ ( ε, x ) of a noise variable ε such that

Original: [Equation]
Revised: [Equation]

Original: The reparameterized ELBO, for which the expectation can be computed, e.g. using Monte Carlo sampling, is then given by
Revised: The reparameterized ELBO, for which the expectation can be computed, e.g. using Monte Carlo sampling, is then given by

Original: [Equation]
Revised: [Equation]

Original: More details on VAE as introduced by Kingma & Welling (2014) are given in appendix A.
Revised: More details on VAE as introduced by Kingma & Welling (2014) are given in appendix A.

Original: 3. L E M O NAD E REPARAMETRIZATION TRICK AND LOSS FUNCTION
Revised: 3. L E M O NAD E REPARAMETRIZATION TRICK AND LOSS FUNCTION

Original: In our case, however, by sampling from Bernoulli distributions we have added discrete stochastic nodes to our computational graph, and we need to ﬁnd differentiable reparameterizations of these nodes.
Revised: In our case, however, by sampling from Bernoulli distributions we have added discrete stochastic nodes to our computational graph, and we need to ﬁnd differentiable reparameterizations of these nodes.

Original: The Bernoulli distribution can be reparameterized using the Gumbel-max trick (Luce, 1959;
Revised: The Bernoulli distribution can be reparameterized using the Gumbel-max trick (Luce, 1959;

Original: Yellott, 1977; Papandreou & Yuille, 2011; Hazan & Jaakkola, 2012; Maddison et al., 2014).
Revised: Yellott, 1977; Papandreou & Yuille, 2011; Hazan & Jaakkola, 2012; Maddison et al., 2014).

Original: This, however, is not differentiable.
Revised: This, however, is not differentiable.

Original: For this reason we use the BinConcrete distribution (Maddison et al., 2016), which is a continuous relaxation of the Bernoulli distribution with temperature parameter λ .
Revised: For this reason we use the BinConcrete distribution (Maddison et al., 2016), which is a continuous relaxation of the Bernoulli distribution with temperature parameter λ .

Original: For λ → 0 the BinConcrete distribution smoothly anneals to the Bernoulli distribution.
Revised: For λ → 0 the BinConcrete distribution smoothly anneals to the Bernoulli distribution.

Original: The
Revised: The

Original: BinConcrete distribution can be reparameterized using the Gumbel-softmax trick (Maddison et al., 2016; Jang et al., 2017), which is differentiable.
Revised: BinConcrete distribution can be reparameterized using the Gumbel-softmax trick (Maddison et al., 2016; Jang et al., 2017), which is differentiable.

Original: Maddison et al.
Revised: Maddison et al.

Original: (2016) show that for a discrete random variable z ∼ Bernoulli ( α ) , the reparameterization of the BinConcrete relaxation of this discrete distribution is
Revised: (2016) show that for a discrete random variable z ∼ Bernoulli ( α ) , the reparameterization of the BinConcrete relaxation of this discrete distribution is

Original: [Equation]
Revised: [Equation]

Original: where U ∼ Uni (0 , 1) and ˜ α
Revised: where U ∼ Uni (0 , 1) and ˜ α

Original: = α/ (1 − α ) .
Revised: = α/ (1 − α ) .

Original: Hence the relaxed and reparameterized lower bound ˜ L ( θ, ˜ α ; x ) ≈ L ( θ, φ ; x ) can be written as
Revised: Hence the relaxed and reparameterized lower bound ˜ L ( θ, ˜ α ; x ) ≈ L ( θ, φ ; x ) can be written as

Original: [Equation]
Revised: [Equation]

Original: where g ˜ α,λ 1 ( y | x ) is the reparameterized BinConcrete relaxation of the variational posterior q φ ( z | x ) and f ˜ a,λ 2 ( y ) the reparameterized relaxation of the prior p a ( z ) .
Revised: where g ˜ α,λ 1 ( y | x ) is the reparameterized BinConcrete relaxation of the variational posterior q φ ( z | x ) and f ˜ a,λ 2 ( y ) the reparameterized relaxation of the prior p a ( z ) .

Original: λ 

Revised: λ

Original: and λ 2 are the respective temperatures and ˜ α and ˜ a the respective locations of the relaxed and reparameterized variational posterior and prior distribution.
Revised: and λ 2 are the respective temperatures and ˜ α and ˜ a the respective locations of the relaxed and reparameterized variational posterior and prior distribution.

Original: The ﬁrst term on the RHS of eq.
Revised: The ﬁrst term on the RHS of eq.

Original: (6) is a negative reconstruction error, showing the connection to traditional autoencoders, while the KL-divergence acts as a regularizer on the approximate posterior q φ ( z | x ) .
Revised: (6) is a negative reconstruction error, showing the connection to traditional autoencoders, while the KL-divergence acts as a regularizer on the approximate posterior q φ ( z | x ) .

Original: As shown in Higgins et al.
Revised: As shown in Higgins et al.

Original: (2017), we can add a β -coefﬁcient to this KL-term which allows to vary the strength of the constraint on the latent space.
Revised: (2017), we can add a β -coefﬁcient to this KL-term which allows to vary the strength of the constraint on the latent space.

Original: Instead of maximizing the lower bound, we will minimize the corresponding loss function 
Revised: Instead of maximizing the lower bound, we will minimize the corresponding loss function (cid:96)

Original: [Equation]
Revised: [Equation]

Original:  with MSE ( x , x (cid:48) ) being the mean-squared error between x and x (cid:48) , and the β -coefﬁcient β KL .
Revised: [Equation] with MSE ( x , x (cid:48) ) being the mean-squared error between x and x (cid:48) , and the β -coefﬁcient β KL .

Original: Datasets with low SNR and large background ﬂuctuations will need a stronger regularization on the activations and hence a larger β KL than higher quality recordings.
Revised: Datasets with low SNR and large background ﬂuctuations will need a stronger regularization on the activations and hence a larger β KL than higher quality recordings.

Original: Hence, adding the β -coefﬁcient to the loss function enables our method to adapt better to the properties of speciﬁc datasets and recording methods.
Revised: Hence, adding the β -coefﬁcient to the loss function enables our method to adapt better to the properties of speciﬁc datasets and recording methods.

Original: 3. L E M O NAD E NETWORK ARCHITECTURE
Revised: 3. L E M O NAD E NETWORK ARCHITECTURE

Original: The encoder network starts with a few convolutional layers with small 2D ﬁlters operating on each frame of the video separately, inspired by the architecture used in Apthorpe et al.
Revised: The encoder network starts with a few convolutional layers with small 2D ﬁlters operating on each frame of the video separately, inspired by the architecture used in Apthorpe et al.

Original: (2016) to extract cells from calcium imaging data.
Revised: (2016) to extract cells from calcium imaging data.

Original: Afterwards the feature maps of the whole video are passed through a ﬁnal convolutional layer with 3D ﬁlters.
Revised: Afterwards the feature maps of the whole video are passed through a ﬁnal convolutional layer with 3D ﬁlters.

Original: These ﬁlters have the size of the feature maps obtained from the single images times a temporal component of length F , which is the expected maximum temporal length of the motifs.
Revised: These ﬁlters have the size of the feature maps obtained from the single images times a temporal component of length F , which is the expected maximum temporal length of the motifs.

Original: We apply padding in the temporal domain to also capture motifs correctly which are cut off at the beginning or end of the analyzed image sequence.
Revised: We apply padding in the temporal domain to also capture motifs correctly which are cut off at the beginning or end of the analyzed image sequence.

Original: The output of the encoder are the parameters ˜
Revised: The output of the encoder are the parameters ˜

Original: α which we need for the reparametrization in eq.
Revised: α which we need for the reparametrization in eq.

Original: From the reparametrization we gain the activations z which are then passed to the decoder.
Revised: From the reparametrization we gain the activations z which are then passed to the decoder.

Original: The decoder consists of a single deconvolution layer with M ﬁlters of the original frame size times the expected motif length F , enforcing the reconstructed data x (cid:48) to be an additive mixture of the decoder ﬁlters.
Revised: The decoder consists of a single deconvolution layer with M ﬁlters of the original frame size times the expected motif length F , enforcing the reconstructed data x (cid:48) to be an additive mixture of the decoder ﬁlters.

Original: Hence, after minimizing the loss the ﬁlters of the decoder contain the detected motifs.
Revised: Hence, after minimizing the loss the ﬁlters of the decoder contain the detected motifs.

Original: Performing these steps on the whole video would be computationally very costly.
Revised: Performing these steps on the whole video would be computationally very costly.

Original: For this reason, we perform each training epoch only on a small subset of the video.
Revised: For this reason, we perform each training epoch only on a small subset of the video.

Original: The subset consists of a few hundred consecutive frames, where the starting point of this short sequence is randomly chosen in each epoch.
Revised: The subset consists of a few hundred consecutive frames, where the starting point of this short sequence is randomly chosen in each epoch.

Original: We found that doing so did not negatively affect the performance of the algorithm.
Revised: We found that doing so did not negatively affect the performance of the algorithm.

Original: By using this strategy we are able to analyse videos of arbitrary length in a computationally efﬁcient way.
Revised: By using this strategy we are able to analyse videos of arbitrary length in a computationally efﬁcient way.

Original: More implementation details can be found in appendix B.
Revised: More implementation details can be found in appendix B.

Original:  E XPERIMENTS AND R ESULTS 4. S YNTHETIC DATA
Revised:  E XPERIMENTS AND R ESULTS 4. S YNTHETIC DATA

Original: The existence of neuronal assemblies is still ﬁercely debated and their detection would only be possible with automated, speciﬁcally tailored tools, like the one proposed in this paper.
Revised: The existence of neuronal assemblies is still ﬁercely debated and their detection would only be possible with automated, speciﬁcally tailored tools, like the one proposed in this paper.

Original: For this reason, no ground truth exists for the identiﬁcation of spatio-temporal motifs in real neurophysiological spike data.
Revised: For this reason, no ground truth exists for the identiﬁcation of spatio-temporal motifs in real neurophysiological spike data.

Original: In order to yet report quantitative accuracies, we test the algorithm on synthetically generated datasets for which ground truth is available.
Revised: In order to yet report quantitative accuracies, we test the algorithm on synthetically generated datasets for which ground truth is available.

Original: For the data generation we used a procedure analogous to the one used in Diego et al.
Revised: For the data generation we used a procedure analogous to the one used in Diego et al.

Original: and Diego & Hamprecht (2013) for testing automated pipelines for the analysis and identiﬁcation of neuronal activity from calcium imaging data.
Revised: and Diego & Hamprecht (2013) for testing automated pipelines for the analysis and identiﬁcation of neuronal activity from calcium imaging data.

Original: In contrast to them, we include neuronal assemblies with temporal ﬁring structure.
Revised: In contrast to them, we include neuronal assemblies with temporal ﬁring structure.

Original: The cells within an assembly can have multiple spikes in a randomly chosen but ﬁxed motif of temporal length up to 30 frames.
Revised: The cells within an assembly can have multiple spikes in a randomly chosen but ﬁxed motif of temporal length up to 30 frames.

Original: We useddifferent assemblies in each sequence.
Revised: We useddifferent assemblies in each sequence.

Original: Additionally, spurious spikes of single neurons were added to simulate noise.
Revised: Additionally, spurious spikes of single neurons were added to simulate noise.

Original: The ratio of spurious spikes to all spikes in the dataset was varied from 0% up to 90% in ten steps.
Revised: The ratio of spurious spikes to all spikes in the dataset was varied from 0% up to 90% in ten steps.

Original: The details of the synthetic data generation can be found in appendix C.1.
Revised: The details of the synthetic data generation can be found in appendix C.1.

Original: To the best of our knowledge, the proposed method is the ﬁrst ever to detect video motifs with temporal structure directly in calcium imaging data.
Revised: To the best of our knowledge, the proposed method is the ﬁrst ever to detect video motifs with temporal structure directly in calcium imaging data.

Original: As a consequence, there are no existing baselines to compare to.
Revised: As a consequence, there are no existing baselines to compare to.

Original: Hence we here propose and evaluate the SCC method presented in Peter et al.
Revised: Hence we here propose and evaluate the SCC method presented in Peter et al.

Original: [Figure] as a baseline.
Revised: (2017) as a baseline.

Original: The SCC algorithm is able to identify motifs with temporal structure in spike trains or calcium transients.
Revised: The SCC algorithm is able to identify motifs with temporal structure in spike trains or calcium transients.

Original: To apply it to our datasets, we ﬁrst have to extract the calcium transients of the individual cells.
Revised: To apply it to our datasets, we ﬁrst have to extract the calcium transients of the individual cells.

Original: For the synthetically generated data we know the location of each cell by construction, so this is possible with arbitrary accuracy.
Revised: For the synthetically generated data we know the location of each cell by construction, so this is possible with arbitrary accuracy.

Original: The output of the SCC algorithm is a matrix that contains for each cell the ﬁring behavior over time within the motif.
Revised: The output of the SCC algorithm is a matrix that contains for each cell the ﬁring behavior over time within the motif.

Original: For a fair comparison we brought the motifs found with LeMoNADe, which are short video sequences, into the same format.
Revised: For a fair comparison we brought the motifs found with LeMoNADe, which are short video sequences, into the same format.

Original: The performance of the algorithms is measured by computing the cosine similarity (Singhal, 2001) between ground truth motifs and detected motifs.
Revised: The performance of the algorithms is measured by computing the cosine similarity (Singhal, 2001) between ground truth motifs and detected motifs.

Original: The cosine similarity is one for identical and zero for orthogonal patterns.
Revised: The cosine similarity is one for identical and zero for orthogonal patterns.

Original: Not all ground truth motifs extend across all 30 frames , and may have almost vanishing luminosity in the last frames.
Revised: Not all ground truth motifs extend across all 30 frames , and may have almost vanishing luminosity in the last frames.

Original: Hence, the discovered motifs can be shifted by a few frames and still capture all relevant parts of the motifs.
Revised: Hence, the discovered motifs can be shifted by a few frames and still capture all relevant parts of the motifs.

Original: For this reason we computed the similarity for the motifs with all possible temporal shifts and took the maximum.
Revised: For this reason we computed the similarity for the motifs with all possible temporal shifts and took the maximum.

Original: More details on the computation of the similarity measure can be found in appendix C.2.
Revised: More details on the computation of the similarity measure can be found in appendix C.2.

Original: We ran both methods on 200 synthetically generated datasets with the parameters shown in table 3 in the appendix.
Revised: We ran both methods on 200 synthetically generated datasets with the parameters shown in table 3 in the appendix.

Original: We here show the results with the correct number of motifs ( M = 3 ) used in both methods.
Revised: We here show the results with the correct number of motifs ( M = 3 ) used in both methods.

Original: In appendix E.1 we show that if the number of motifs is overestimated (here M > 3 ),
Revised: In appendix E.1 we show that if the number of motifs is overestimated (here M > 3 ),

Original: LeMoNADe still identiﬁes the correct motifs, but they are repeated multiple times in the surplus ﬁlters.
Revised: LeMoNADe still identiﬁes the correct motifs, but they are repeated multiple times in the surplus ﬁlters.

Original: Hence this does not reduce the performance of the algorithm.
Revised: Hence this does not reduce the performance of the algorithm.

Original: The temporal extent of the motifs was set to F = 31 to give the algorithms the chance to also capture the longer patterns.
Revised: The temporal extent of the motifs was set to F = 31 to give the algorithms the chance to also capture the longer patterns.

Original: The cosine similarity of the found motifs to the set of ground truth motifs, averaged over all found motifs and all experiments for each of the ten noise levels, is shown in ﬁgure 4.
Revised: The cosine similarity of the found motifs to the set of ground truth motifs, averaged over all found motifs and all experiments for each of the ten noise levels, is shown in ﬁgure 4.

Original: The results in ﬁgure 4 show that LeMoNADe performs as well as SCC in detecting motifs and also shows a similar stability in the presence of noise as SCC.
Revised: The results in ﬁgure 4 show that LeMoNADe performs as well as SCC in detecting motifs and also shows a similar stability in the presence of noise as SCC.

Original: This is surprising since LeMoNADe does not need the previous extraction of individual cells and hence has to solve a much harder problem than SCC.
Revised: This is surprising since LeMoNADe does not need the previous extraction of individual cells and hence has to solve a much harder problem than SCC.

Original: In order to verify that the results achieved by LeMoNADe and SCC range signiﬁcantly above chance, we performed a bootstrap (BS) test.
Revised: In order to verify that the results achieved by LeMoNADe and SCC range signiﬁcantly above chance, we performed a bootstrap (BS) test.

Original: For this, multiple datasets were created with similar spike distributions as before, but with no reoccurring motif-like ﬁring patterns.
Revised: For this, multiple datasets were created with similar spike distributions as before, but with no reoccurring motif-like ﬁring patterns.

Original: We compiled a distribution of similarities between patterns suggested by the proposed method and randomly sampled segments of same length and general statistics from that same BS dataset.
Revised: We compiled a distribution of similarities between patterns suggested by the proposed method and randomly sampled segments of same length and general statistics from that same BS dataset.

Original: The full BS distributions are shown in appendix C.3.
Revised: The full BS distributions are shown in appendix C.3.

Original: The 95%-tile of the BS distributions for each noise level are also shown in ﬁgure 4.
Revised: The 95%-tile of the BS distributions for each noise level are also shown in ﬁgure 4.

Original: Figure 5 shows an exemplary result from one of the analysed synthetic datasets with 10% noise and maximum temporal extend of the ground truth motifs of 28 frames.
Revised: Figure 5 shows an exemplary result from one of the analysed synthetic datasets with 10% noise and maximum temporal extend of the ground truth motifs of 28 frames.

Original: All three motifs were correctly identiﬁed (see ﬁgure 5a) with a small temporal shift.
Revised: All three motifs were correctly identiﬁed (see ﬁgure 5a) with a small temporal shift.

Original: This shift does not reduce the performance as it is compensated by a corresponding shift in the activations of the motifs (see ﬁgure 5b).
Revised: This shift does not reduce the performance as it is compensated by a corresponding shift in the activations of the motifs (see ﬁgure 5b).

Original: In order to show that the temporal structure of the found motifs matches the ground truth, in ﬁgure 5a for motif 1 and 2 we corrected the shift of one and two frames, respectively.
Revised: In order to show that the temporal structure of the found motifs matches the ground truth, in ﬁgure 5a for motif 1 and 2 we corrected the shift of one and two frames, respectively.

Original: We also show the results after extracting the individual cells from the motifs and the results from SCC in ﬁgure 5c.
Revised: We also show the results after [Figure] [Figure]

Original: [-One can see that the results are almost identical, again except for small temporal shifts. frameframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframe 1800 0 a c t i v a t i o n m o t i f640 650         1800 
-]
Revised: [-One can see that the results are almost identical, again except for small temporal shifts. frameframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframeframe 1800 0 a c t i v a t i o n m o t i f640 650         1800 
-][Figure]

Original: a c t i v a t
Revised: a c t i v a t

Original: i o n m o t i [-f1640 
-]
Revised: i o n m o t i [-f1640 
-]f

Original: [Figure]
Revised: [Figure]

Original: 100a c t i v a t i o nm o t
Revised: c t i v a t i o n

Original: [-i f820 
-]
Revised: [-i f820 
-][Figure]

Original: 1 6 11 16 21 26 31 010 15 n e u r o n motif1 6 11 16 21 26 31 010motif1 6 11 16 21 26 31 010motif1 6 11 16 21 26 31 010 15 n e u r o n[-1 6 11 16 21 26 31 0101 6 11 16 21 26 31 0101 6 11 16 21 
-]
Revised: t i v a t i o n[-1 6 11 16 21 26 31 0101 6 11 16 21 26 31 0101 6 11 16 21 
-]

Original: [Figure]
Revised: [Figure]

Original: 010 15 n e u r o [-n 1 6 11 16 
-]
Revised: o [-n 1 6 11 16 
-]t i

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: {+21 26 31 010 15 n e u r o n motif1 6 11 16 21 26 31 010motif1 6 11 16 21 26 31 010motif1 6 11 16 21 26 31 010 15 n e u r o n 1 6 11 16 21 26 31 0101 6 11 16 21 26 31 0101 6 11 16 21 26 31 frame 010 15 n e u r o n 1 6 11 16 21 26 31 frame 0101 6 11 16 21 26 31 frame 
+}
Revised: {+21 26 31 010 15 n e u r o n motif1 6 11 16 21 26 31 010motif1 6 11 16 21 26 31 010motif1 6 11 16 21 26 31 010 15 n e u r o n 1 6 11 16 21 26 31 0101 6 11 16 21 26 31 0101 6 11 16 21 26 31 frame 010 15 n e u r o n 1 6 11 16 21 26 31 frame 0101 6 11 16 21 26 31 frame 
+}

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: match ground truth for better readability motif sequences can be appendix E
Revised: extracting the individual cells from the motifs and the results from SCC in ﬁgure 5c. One can see that the results are almost identical, again except for small temporal shifts.

Original: 4. R EAL DATA
Revised: 4. R EAL DATA

Original: We applied the proposed method on two datasets obtained from organotypic hippocampal slice cultures.
Revised: We applied the proposed method on two datasets obtained from organotypic hippocampal slice cultures.

Original: The cultures were prepared from 7–9-day-old Wistar rats as described in Kann et al.
Revised: The cultures were prepared from 7–9-day-old Wistar rats as described in Kann et al.

Original: (2003) and Schneider et al.
Revised: (2003) and Schneider et al.

Original: The ﬂuorescent Ca 2+ sensor, GCaMP6f
Revised: The ﬂuorescent Ca 2+ sensor, GCaMP6f

Original: (Chen et al., 2013), was delivered to the neurons by an adeno-associated virus (AAV).
Revised: (Chen et al., 2013), was delivered to the neurons by an adeno-associated virus (AAV).

Original: Neurons in stratum pyramidale of CA3 were imaged for 6.5 (dataset 1) and 5 minutes (dataset 2) in the presence of the cholinergic receptor agonist carbachol.
Revised: Neurons in stratum pyramidale of CA3 were imaged for 6.5 (dataset 1) and 5 minutes (dataset 2) in the presence of the cholinergic receptor agonist carbachol.

Original: For more details on the generation of these datasets see appendix D.1. 
Revised: For more details on the generation of these datasets see appendix D.1. D ATASET [Figure] [Figure]

Original: The proposed method was run on these datasets with the parameter settings shown in table 3 in the appendix E, where we also provide additional comments on the parameter settings.
Revised: D ATASET [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: 2a c t i v a t i o n

Revised: [Figure]

Revised: o t i f

Revised: [Figure]

Revised: [Figure]

Revised: 1200200a c t i v a t i o n m o t

Revised: i f      1200 040a c t i v a t

Revised: i o n m o t i f

Revised: [Figure]

Revised: 2a c t i v a t i o n m o t

Revised: i f

Revised: [Figure]

Revised: The proposed method was run on these datasets with the parameter settings shown in table 3 in the appendix E, where we also provide additional comments on the parameter settings.

Original: The analysis of the datasets took less than two hours on a Ti 1080 GPU.
Revised: The analysis of the datasets took less than two hours on a Ti 1080 GPU.

Original: Before running the analysis we computed ∆ F/F for the datasets.
Revised: Before running the analysis we computed ∆ F/F for the datasets.

Original: We looked for up to three motifs with a maximum extent of F = 21 frames.
Revised: We looked for up to three motifs with a maximum extent of F = 21 frames.

Original: The results are shown in ﬁgure 6.
Revised: The results are shown in ﬁgure 6.

Original: For both datasets, one motif in ﬁgure 6a consists of multiple cells, shows repeated activation over the recording period (see ﬁgure 6b, 6c), and contains temporal structure (see ﬁgure 6d).
Revised: For both datasets, one motif in ﬁgure 6a consists of multiple cells, shows repeated activation over the recording period (see ﬁgure 6b, 6c), and contains temporal structure (see ﬁgure 6d).

Original: The other two “motifs” can easily be identiﬁed as artefacts and background ﬂuctuations.
Revised: The other two “motifs” can easily be identiﬁed as artefacts and background ﬂuctuations.

Original: As SCC and many other motif detection methods, LeMoNADe suffers from the fact that such artefacts, especially single events with extremely high neuronal activation, potentially explain a large part of the data and hence can be falsely detected as motifs.
Revised: As SCC and many other motif detection methods, LeMoNADe suffers from the fact that such artefacts, especially single events with extremely high neuronal activation, potentially explain a large part of the data and hence can be falsely detected as motifs.

Original: Nevertheless, these events can be easily identiﬁed by simply looking at the motif videos or thresholding the activations as done in ﬁgure 6c.
Revised: Nevertheless, these events can be easily identiﬁed by simply looking at the motif videos or thresholding the activations as done in ﬁgure 6c.

Original: Although the found motifs also include neuropil activation, this does not imply this was indeed used by the VAE as a deﬁning feature of the motifs, just that it was also present in the images.
Revised: Although the found motifs also include neuropil activation, this does not imply this was indeed used by the VAE as a deﬁning feature of the motifs, just that it was also present in the images.

Original: Dendritic/axonal structures are part of the activated neurons and therefore also visible in the motif videos.
Revised: Dendritic/axonal structures are part of the activated neurons and therefore also visible in the motif videos.

Original: If necessary, these structures can be removed by post-processing steps.
Revised: If necessary, these structures can be removed by post-processing steps.

Original: As
Revised: As

Original: LeMoNADe reduces the problem to the short motif videos instead of the whole calcium imaging video, the neuropil subtraction becomes much more feasible.D ATASET [Figure] [Figure]
Revised: LeMoNADe reduces the problem to the short motif videos instead of the whole calcium imaging video, the neuropil subtraction becomes much more feasible.

Original: [Figure]

Original: D ATASET

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: 2a c t i v a t i o n

Original: [Figure]

Original: t i f

Original: [Figure]

Original: 1200200a c t i v a t i o n m o t

Original: i f      1200 040a c t i v a t

Original: i o n m o t i f

Original: [Figure]

Original: 2a c t i v a t i o n m o t

Original: i f

Original: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have presented a novel approach for the detection of neuronal assemblies that directly operates on the calcium imaging data, making the cumbersome extraction of individual cells and discrete spike times from the raw data dispensable.
Revised: We have presented a novel approach for the detection of neuronal assemblies that directly operates on the calcium imaging data, making the cumbersome extraction of individual cells and discrete spike times from the raw data dispensable.

Original: The motifs are extracted as short, repeating image sequences.
Revised: The motifs are extracted as short, repeating image sequences.

Original: This provides them in a very intuitive way and additionally returns information about the spatial distribution of the cells within an assembly.
Revised: This provides them in a very intuitive way and additionally returns information about the spatial distribution of the cells within an assembly.

Original: The proposed method’s performance in identifying motifs is equivalent to that of a state-of-the-art method that requires the previous extraction of individual cells.
Revised: The proposed method’s performance in identifying motifs is equivalent to that of a state-of-the-art method that requires the previous extraction of individual cells.

Original: Moreover, we were able to identify repeating ﬁring patterns in two datasets from hippocampal slice cultures, proving that the method is capable of handling real calcium imaging conditions.
Revised: Moreover, we were able to identify repeating ﬁring patterns in two datasets from hippocampal slice cultures, proving that the method is capable of handling real calcium imaging conditions.

Original: For future work, a post-processing step as used in Peter et al.
Revised: For future work, a post-processing step as used in Peter et al.

Original: (2017) or a group sparsity regularization similar to the ones used in Bascol et al.
Revised: (2017) or a group sparsity regularization similar to the ones used in Bascol et al.

Original: (2016) or Mackevicius et al.
Revised: (2016) or Mackevicius et al.

Original: (2018) could be added to determine a plausible number of motifs automatically.
Revised: (2018) could be added to determine a plausible number of motifs automatically.

Original: Moreover, additional latent dimensions could be introduced to capture artefacts and background ﬂuctuations and hence automatically separate them from the actual motifs.
Revised: Moreover, additional latent dimensions could be introduced to capture artefacts and background ﬂuctuations and hence automatically separate them from the actual motifs.

Original: The method is expected to, in principle, also work on other functional imaging modalities.
Revised: The method is expected to, in principle, also work on other functional imaging modalities.

Original: We will investigate the possibility of detecting motifs using LeMoNADe on recordings from human fMRI or voltage-sensitive dyes in the future.
Revised: We will investigate the possibility of detecting motifs using LeMoNADe on recordings from human fMRI or voltage-sensitive dyes in the future

Revised: A CKNOWLEDGMENTS

Revised: EK thanks Ferran Diego for sharing his knowledge on generating synthetic data and for his scientiﬁc advice.

Revised: DD acknowledges partial ﬁnancial support by DFG Du 354/8-1.

Revised: EK, HS, JS, SE, OK, DD and FAH gratefully acknowledge partial ﬁnancial support by DFG SFB 1134.

Original: A PPENDIX A V ARIATIONAL AUTOENCODER
Revised: A PPENDIX A V ARIATIONAL AUTOENCODER

Original: Variational autoencoder (VAE) are generative latent variable models which were ﬁrst described in Kingma & Welling (2014).
Revised: Variational autoencoder (VAE) are generative latent variable models which were ﬁrst described in Kingma & Welling (2014).

Original: The data x = (cid:8) x ( i ) (cid:9) N i =1 , consisting of N samples of some random variable x , is generated by ﬁrst drawing a latent variable z ( i ) from a prior distribution p ( z ) and then sampling from the conditional distribution p θ ∗ ( x | z ) with parameters θ ∗ .
Revised: The data x = (cid:8) x ( i ) (cid:9) N i =1 , consisting of N samples of some random variable x , is generated by ﬁrst drawing a latent variable z ( i ) from a prior distribution p ( z ) and then sampling from the conditional distribution p θ ∗ ( x | z ) with parameters θ ∗ .

Original: The distribution p θ ∗ ( x | z ) belongs to the parametric family p θ ( x | z ) with differentiable PDFs w.r.t.
Revised: The distribution p θ ∗ ( x | z ) belongs to the parametric family p θ ( x | z ) with differentiable PDFs w.r.t.

Original: θ and z .
Revised: θ and z .

Original: Both the true parameters θ ∗ as well as the latent variables
Revised: Both the true parameters θ ∗ as well as the latent variables

Original: z
Revised: z

Original: ( i ) are unknown.
Revised: ( i ) are unknown.

Original: We are interested in an approximate posterior inference of the latent variables z given some data x .
Revised: We are interested in an approximate posterior inference of the latent variables z given some data x .

Original: The true posterior p θ ( z | x ) , however, is usually intractable.
Revised: The true posterior p θ ( z | x ) , however, is usually intractable.

Original: But it can be approximated by introducing the recognition model (or approximate posterior) q φ ( z | x ) .
Revised: But it can be approximated by introducing the recognition model (or approximate posterior) q φ ( z | x ) .

Original: We want to learn both the recognition model parameters φ as well as the generative model parameters θ .
Revised: We want to learn both the recognition model parameters φ as well as the generative model parameters θ .

Original: The recognition model is usually referred to as the probabilistic encoder and p θ ( x | z ) is called the probabilistic decoder.
Revised: The recognition model is usually referred to as the probabilistic encoder and p θ ( x | z ) is called the probabilistic decoder.

Original: In order to learn the variational parameters φ we want to minimise the KL-divergence between approximate and true posterior KL ( q φ ( z | x ) (cid:107) p θ ( z | x )) .
Revised: In order to learn the variational parameters φ we want to minimise the KL-divergence between approximate and true posterior KL ( q φ ( z | x ) (cid:107) p θ ( z | x )) .

Original: Therefore we use the fact that the marginal likelihood p θ ( x ) can be written as
Revised: Therefore we use the fact that the marginal likelihood p θ ( x ) can be written as

Original: [Equation]
Revised: [Equation]

Original: As the KL-divergence is non-negative, we can minimize KL (cid:0) q φ ( z | x ) (cid:107) p θ ( z | x ) (cid:1) by maximizing the (variational) lower bound L ( p, q ; x ) with 
Revised: As the KL-divergence is non-negative, we can minimize KL (cid:0) q φ ( z | x ) (cid:107) p θ ( z | x ) (cid:1) by maximizing the (variational) lower bound L ( p, q ; x ) with L ( p

Original: [Equation]
Revised: [Equation]

Original: In order to optimise the lower bound L ( p, q ; x ) w.r.t.
Revised: In order to optimise the lower bound L ( p, q ; x ) w.r.t.

Original: both the variational parameters φ and the generative parameters θ , we need to compute the gradients
Revised: both the variational parameters φ and the generative parameters θ , we need to compute the gradients

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst part of the lower bound the gradient w.r.t.
Revised: For the ﬁrst part of the lower bound the gradient w.r.t.

Original: θ can be easily computed using Monte Carlo sampling
Revised: θ can be easily computed using Monte Carlo sampling

Original: [Equation]
Revised: [Equation]

Original: with z s ∼ q φ ( z | x ) .
Revised: with z s ∼ q φ ( z | x ) .

Original: The gradient w.r.t.
Revised: The gradient w.r.t.

Original: φ , however, does not take the form of an expectation in z and can therefore not be sampled that easily:
Revised: φ , however, does not take the form of an expectation in z and can therefore not be sampled that easily:

Original: [Equation]
Revised: [Equation]

Original: However, in most cases we can use the reparameterization trick to overcome this problem: the random variable ˜
Revised: However, in most cases we can use the reparameterization trick to overcome this problem: the random variable ˜

Original: z ∼ q φ ( z | x ) can be reparameterised using a differentiable transformation h φ ( ε, x ) of a noise variable ε such that
Revised: z ∼ q φ ( z | x ) can be reparameterised using a differentiable transformation h φ ( ε, x ) of a noise variable ε such that

Original: [Equation]
Revised: [Equation]

Original: We now can compute the gradient w.r.t.
Revised: We now can compute the gradient w.r.t.

Original: φ again using Monte Carlo sampling
Revised: φ again using Monte Carlo sampling

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with ε s ∼ p ( ε ) .
Revised: with ε s ∼ p ( ε ) .

Original: Hence, the reparameterized lower bound ˜ L ( p, q ; x ) ≈ L ( p, q ; x ) can be written as
Revised: Hence, the reparameterized lower bound ˜ L ( p, q ; x ) ≈ L ( p, q ; x ) can be written as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: with z s = h φ ( ε s , x ) , ε ∼ p ( ε ) .
Revised: with z s = h φ ( ε s , x ) , ε ∼ p ( ε ) .

Original: The ﬁrst term on the RHS of eq.
Revised: The ﬁrst term on the RHS of eq.

Original: (15) is a negative reconstruction error, showing the connection to traditional autoencoders, while the KL-divergence acts as a regularizer on the approximate posterior q φ ( z | x ) .
Revised: (15) is a negative reconstruction error, showing the connection to traditional autoencoders, while the KL-divergence acts as a regularizer on the approximate posterior q φ ( z | x ) .

Original: B L E M O NAD E NETWORK ARCHITECTURE AND IMPLEMENTATION DETAILS B. E NCODER
Revised: B L E M O NAD E NETWORK ARCHITECTURE AND IMPLEMENTATION DETAILS B. E NCODER

Original: The encoder network starts with a few convolutional layers with small 2D ﬁlters operating on each frame of the video separately, inspired by the architecture used in Apthorpe et al.
Revised: The encoder network starts with a few convolutional layers with small 2D ﬁlters operating on each frame of the video separately, inspired by the architecture used in Apthorpe et al.

Original: (2016) to extract cells from calcium imaging data.
Revised: (2016) to extract cells from calcium imaging data.

Original: The details of this network are shown in table 1.
Revised: The details of this network are shown in table 1.

Original: Afterwards the feature maps of the whole video are passed through a ﬁnal convolutional layer with 3D ﬁlters.
Revised: Afterwards the feature maps of the whole video are passed through a ﬁnal convolutional layer with 3D ﬁlters.

Original: These ﬁlters have size of the feature maps gained from the single images times a temporal component of length F , which is the expected maximum temporal extent of a motif.
Revised: These ﬁlters have size of the feature maps gained from the single images times a temporal component of length F , which is the expected maximum temporal extent of a motif.

Original: We use 2 · M ﬁlters and apply padding in the temporal domain to avoid edge effects.
Revised: We use 2 · M ﬁlters and apply padding in the temporal domain to avoid edge effects.

Original: By this also motifs that are cut off at the beginning or the end of the sequence can be captured properly.
Revised: By this also motifs that are cut off at the beginning or the end of the sequence can be captured properly.

Original: The output of the encoder are 2 · M feature maps of size ( T + F − 1) × 1 × 1 .
Revised: The output of the encoder are 2 · M feature maps of size ( T + F − 1) × 1 × 1 .

Original: B. R EPARAMETERIZATION
Revised: B. R EPARAMETERIZATION

Original: Instead of reparameterizing the Bernoulli distributions, we will reparameterize their BinConcrete relaxations.
Revised: Instead of reparameterizing the Bernoulli distributions, we will reparameterize their BinConcrete relaxations.

Original: The BinConcrete relaxation of a Bernoulli distribution with parameter α takes as input parameter ˜
Revised: The BinConcrete relaxation of a Bernoulli distribution with parameter α takes as input parameter ˜

Original: α = α/ (1 − α ) .
Revised: α = α/ (1 − α ) .

Original: Maddison et al.
Revised: Maddison et al.

Original: (2016) showed that instead of using the normalized probabilities α , we can also perform the reparametrization with unnormalized parameters α 1 and α 2 , where α 1 is the probability to sample a one and α 2 is the probability to sample a zero and ˜ α
Revised: (2016) showed that instead of using the normalized probabilities α , we can also perform the reparametrization with unnormalized parameters α 1 and α 2 , where α 1 is the probability to sample a one and α 2 is the probability to sample a zero and ˜ α

Original: =
Revised: =

Original: α 1 /α 2 .
Revised: α 1 /α 2 .

Original: The ﬁrst M feature maps, which were outputted by the encoder, are assigned to contain the unnormalised probabilities α 1 m,t for the activation of motif m in frame t to be one.
Revised: The ﬁrst M feature maps, which were outputted by the encoder, are assigned to contain the unnormalised probabilities α 1 m,t for the activation of motif m in frame t to be one.

Original: The second M feature maps contain the unnormalized probabilities α 2 m,t for the activation of motif m in frame t to be zero.
Revised: The second M feature maps contain the unnormalized probabilities α 2 m,t for the activation of motif m in frame t to be zero.

Original: The parameter ˜ α that is needed for the reparameterized BinConcrete distribution is obtained by dividing the two vectors elementwise: ˜ α mt
Revised: The parameter ˜ α that is needed for the reparameterized BinConcrete distribution is obtained by dividing the two vectors elementwise: ˜ α mt

Original: = α 1 m,t /α 2 m,t .
Revised: = α 1 m,t /α 2 m,t .

Original: We use the reparameterization trick to sample from BinConcrete( ˜ α mt ) as follows:
Revised: We use the reparameterization trick to sample from BinConcrete( ˜ α mt ) as follows:

Original: First we sample (cid:110) { U mt } T + F − 1 t
Revised: First we sample (cid:110) { U mt } T + F − 1 t

Original: = (cid:111)
Revised: = (cid:111)

Original: [Equation]
Revised: [Equation]

Original: m =1 from a uniform distribution Uni (0 , 1) .
Revised: m =1 from a uniform distribution Uni (0 , 1) .

Original: Next, we compute y with
Revised: Next, we compute y with

Original: [Equation]
Revised: [Equation]

Original: Finally, we gain z according to
Revised: Finally, we gain z according to

Original: [Equation]
Revised: [Equation]

Original: for all m = 1 , . . .
Revised: for all m = 1 , . . .

Original: , M and t = 1 , . .
Revised: , M and t = 1 , . .

Original: , T + F − 1 .
Revised: , T + F − 1 .

Original: The multiplication by α 1 m,t in eq.
Revised: The multiplication by α 1 m,t in eq.

Original: (17) is not part of the original reparametrization trick (Maddison et al., 2016; Jang et al., 2017).
Revised: (17) is not part of the original reparametrization trick (Maddison et al., 2016; Jang et al., 2017).

Original: But we found that the results of the algorithm improved dramatically as we scaled the activations with the α 1 -values that were originally predicted from the encoder network.
Revised: But we found that the results of the algorithm improved dramatically as we scaled the activations with the α 1 -values that were originally predicted from the encoder network.

Original: B. D ECODER
Revised: B. D ECODER

Original: The input to the decoder are now the activations z .
Revised: The input to the decoder are now the activations z .

Original: The decoder consists of a single deconvolution layer with M ﬁlters of the original frame size times the expected motif length F .
Revised: The decoder consists of a single deconvolution layer with M ﬁlters of the original frame size times the expected motif length F .

Original: These deconvolution ﬁlters contain the motifs we are looking for.
Revised: These deconvolution ﬁlters contain the motifs we are looking for.

Original: The details of the used networks as well as the sizes of the inputs and outputs of the different steps are shown in table 1.
Revised: The details of the used networks as well as the sizes of the inputs and outputs of the different steps are shown in table 1.

Original: Algorithm 1 summarizes the reparametrization and updates.
Revised: Algorithm 1 summarizes the reparametrization and updates.

Original: C E XPERIMENTS AND RESULTS ON SYNTHETIC DATA C. S YNTHETIC DATA GENERATION
Revised: C E XPERIMENTS AND RESULTS ON SYNTHETIC DATA C. S YNTHETIC DATA GENERATION

Original: We created 200 artiﬁcial sequences of length 60 s with a frame rate of 30 fps and 128 × 128 pixel per image.
Revised: We created 200 artiﬁcial sequences of length 60 s with a frame rate of 30 fps and 128 × 128 pixel per image.

Original: The number of cells was varied and they were located randomly in the image plane with an overlap of up to 30 % .
Revised: The number of cells was varied and they were located randomly in the image plane with an overlap of up to 30 % .

Original: The cell shapes were selected randomly from 36 shapes extracted from
Revised: The cell shapes were selected randomly from 36 shapes extracted from

Original: [Table]
Revised: [Table]

Original: Input: raw video x , normalized to zero mean and unit variance, architectures f θ , α φ , hyperparameter λ 1 , λ 2 , ˜ a, β KL
Revised: Input: raw video x , normalized to zero mean and unit variance, architectures f θ , α φ , hyperparameter λ 1 , λ 2 , ˜ a, β KL

Original: Result: trained f θ , α φ θ, φ ← Initialize network parameters repeat //
Revised: Result: trained f θ , α φ θ, φ ← Initialize network parameters repeat //

Original: Sample subset of video x sub ← Randomly chosen sequence of consecutive frames from x // Encoding step
Revised: Sample subset of video x sub ← Randomly chosen sequence of consecutive frames from x // Encoding step

Original: Encode x sub to get ˜ α as described in section B.1 and B.
Revised: Encode x sub to get ˜ α as described in section B.1 and B.

Original: // Latent Step
Revised: // Latent Step

Original: Sample noise U ∼ Uni (0 , 1)
Revised: Sample noise U ∼ Uni (0 , 1)

Original: Compute y following eq.
Revised: Compute y following eq.

Original: Compute z following eq.
Revised: Compute z following eq.

Original: Decoding Step x (cid:48) sub ← decode via f θ ( z ) //
Revised: Decoding Step x (cid:48) sub ← decode via f θ ( z ) //

Original: Update Parameters
Revised: Update Parameters

Original: Compute gradients of loss φ, θ ← update via ∇ φ,θ (cid:96) ( x sub , x (cid:48) sub , ˜ α, λ 1 , ˜ a, λ 2 , β KL ) (see eq.
Revised: Compute gradients of loss φ, θ ← update via ∇ φ,θ (cid:96) ( x sub , x (cid:48) sub , ˜ α, λ 1 , ˜ a, λ 2 , β KL ) (see eq.

Original: (7) in the main paper) until until convergence of θ, φ ; real data.
Revised: (7) in the main paper) until until convergence of θ, φ ; real data.

Original: The transients were modelled as two-sided exponential decay with scales of 50 ms and 400 ms , respectively.
Revised: The transients were modelled as two-sided exponential decay with scales of 50 ms and 400 ms , respectively.

Original: In contrast to Diego & Hamprecht (2013), we included neuronal assemblies with temporal ﬁring structure.
Revised: In contrast to Diego & Hamprecht (2013), we included neuronal assemblies with temporal ﬁring structure.

Original: That means cells within an assembly can perform multiple spikes in a randomly chosen but ﬁxed motif of temporal length up to 30 frames.
Revised: That means cells within an assembly can perform multiple spikes in a randomly chosen but ﬁxed motif of temporal length up to 30 frames.

Original: We used 3 different assemblies in each sequence.
Revised: We used 3 different assemblies in each sequence.

Original: The assembly activity itself was modelled as a Poisson process (Lopes-dos Santos et al., 2013) with a mean of 0 .
Revised: The assembly activity itself was modelled as a Poisson process (Lopes-dos Santos et al., 2013) with a mean of 0 .

Original: 15 spikes / second and a refractory period of at least the length of the motif itself.
Revised: 15 spikes / second and a refractory period of at least the length of the motif itself.

Original: By construction the cell locations as well as the ﬁring motifs are known for these datasets.
Revised: By construction the cell locations as well as the ﬁring motifs are known for these datasets.

Original: In order to simulate the conditions in real calcium imaging videos as good as possible, we added Gaussian background noise with a relative amplitude ( max intensity − mean intensity ) /σ
Revised: In order to simulate the conditions in real calcium imaging videos as good as possible, we added Gaussian background noise with a relative amplitude ( max intensity − mean intensity ) /σ

Original: noise between 10 and 20.
Revised: noise between 10 and 20.

Original: Additionally, spurious spikes not belonging to any motif were added.
Revised: Additionally, spurious spikes not belonging to any motif were added.

Original: The amount
Revised: The amount

Original: [Table]
Revised: [Table]

Original: of spurious spikes was varied from 0% up to 90% of all spikes in the dataset.
Revised: of spurious spikes was varied from 0% up to 90% of all spikes in the dataset.

Original: For each of the 10 noise levels 20 datasets were generated.
Revised: For each of the 10 noise levels 20 datasets were generated.

Original: C. S IMILARITY MEASURE
Revised: C. S IMILARITY MEASURE

Original: The performance of the algorithms is measured by computing the cosine similarity (Singhal, 2001) between ground truth motifs and found motifs.
Revised: The performance of the algorithms is measured by computing the cosine similarity (Singhal, 2001) between ground truth motifs and found motifs.

Original: The found motifs are in an arbitrary order, not necessarily corresponding to the order of the ground truth motifs.
Revised: The found motifs are in an arbitrary order, not necessarily corresponding to the order of the ground truth motifs.

Original: Additionally, the found motifs can be shifted in time compared to the ground truth.
Revised: Additionally, the found motifs can be shifted in time compared to the ground truth.

Original: To account for this fact, we compute the similarity between the found motifs and each of the ground truth motifs with all possible temporal shifts and take the maximum.
Revised: To account for this fact, we compute the similarity between the found motifs and each of the ground truth motifs with all possible temporal shifts and take the maximum.

Original: Hence, the similarity between the m -th found motif and the set of ground truth motifs G is deﬁned by
Revised: Hence, the similarity between the m -th found motif and the set of ground truth motifs G is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where M m is the m -th found motif, (cid:104)· , ·(cid:105) is the dot product and vec ( · ) vectorizes the motifs with dimensions F × N into a vector of length F · N , where N is the number of cells.
Revised: where M m is the m -th found motif, (cid:104)· , ·(cid:105) is the dot product and vec ( · ) vectorizes the motifs with dimensions F × N into a vector of length F · N , where N is the number of cells.

Original: The shift operator s → ( · ) moves a motif s frames forward in time while keeping the same size and ﬁlling missing values appropriately with zeros (Smaragdis, 2004).
Revised: The shift operator s → ( · ) moves a motif s frames forward in time while keeping the same size and ﬁlling missing values appropriately with zeros (Smaragdis, 2004).

Original: The cosine similarity of the found motifs to the set of ground truth motifs was averaged over all found motifs and all experiments for each noise level.
Revised: The cosine similarity of the found motifs to the set of ground truth motifs was averaged over all found motifs and all experiments for each noise level.

Original: The average similarities achieved with LeMoNADe and SCC as well as the 5% signiﬁcance threshold of the BS distribution for each noise level can be found in table 2.
Revised: The average similarities achieved with LeMoNADe and SCC as well as the 5% signiﬁcance threshold of the BS distribution for each noise level can be found in table 2.

Original: C. B OOTSTRAP - BASED SIGNIFICANCE TEST
Revised: C. B OOTSTRAP - BASED SIGNIFICANCE TEST

Original: Statistical methods for testing for cell assemblies (or spatio-temporal patterns more generally) have been advanced tremendously in recent years, addressing many of the issues that have plagued older approaches (Grün, 2009; Staude et al., 2010a;b; Russo & Durstewitz, 2017).
Revised: Statistical methods for testing for cell assemblies (or spatio-temporal patterns more generally) have been advanced tremendously in recent years, addressing many of the issues that have plagued older approaches (Grün, 2009; Staude et al., 2010a;b; Russo & Durstewitz, 2017).

Original: Simple shufﬂe bootstraps are not necessarily the best methods if they destroy too much of the auto-correlative structure, and they can severely underestimate the distributional tails (Davison et al., 1997).
Revised: Simple shufﬂe bootstraps are not necessarily the best methods if they destroy too much of the auto-correlative structure, and they can severely underestimate the distributional tails (Davison et al., 1997).

Original: Therefore we use sophisticated parametric, model-based bootstraps which retain the full statistical structure of the original data, except for the crucial feature of repeating motifs.
Revised: Therefore we use sophisticated parametric, model-based bootstraps which retain the full statistical structure of the original data, except for the crucial feature of repeating motifs.

Original: In order to provide a ’null hypothesis (H0)’ reference for the motif similarities returned by LeMoNADe (or other methods), we used the following bootstrap (BS) based test procedure: We generated 20 datasets analogue to those described in section C.1, i.e. with same spiking statistics and temporal
Revised: In order to provide a ’null hypothesis (H0)’ reference for the motif similarities returned by LeMoNADe (or other methods), we used the following bootstrap (BS) based test procedure: We generated 20 datasets analogue to those described in section C.1, i.e. with same spiking statistics and temporal

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: 6 0.0 0.2 0.4 0.6 0.8 1.0 similarity 04 0.0 0.2 0.4 0.6 0.8 1.0 similarity 04 6 0.0 0.2 0.4 0.6 0.8 1.0 similarity 010 15 convolution with calcium transients, but without repeating motifs.
Revised: 0.0 0.2 0.4 0.6 0.8 1.0 similarity 04 0.0 0.2 0.4 0.6 0.8 1.0 similarity 04 6 0.0 0.2 0.4 0.6 0.8 1.0 similarity 010 15 convolution with calcium transients, but without repeating motifs.

Original: These motif-less H0 datasets were then processed by LeMoNADe in the very same way as the motif-containing datasets, i.e. with the parameter settings as shown in table 3.
Revised: These motif-less H0 datasets were then processed by LeMoNADe in the very same way as the motif-containing datasets, i.e. with the parameter settings as shown in table 3.

Original: From each of these BS datasets 150 random samples of the same temporal length as that of the ’detected’ motifs were drawn.
Revised: From each of these BS datasets 150 random samples of the same temporal length as that of the ’detected’ motifs were drawn.

Original: For each BS dataset, the similarities between each of the found motifs and all of the 150 random samples were computed as described in section C.2.
Revised: For each BS dataset, the similarities between each of the found motifs and all of the 150 random samples were computed as described in section C.2.

Original: As datasets with higher noise levels have different spiking statistics, we repeated this procedure for each of the ten noise levels.
Revised: As datasets with higher noise levels have different spiking statistics, we repeated this procedure for each of the ten noise levels.

Original: Figure 7 shows the BS distributions (top).
Revised: Figure 7 shows the BS distributions (top).

Original: We also show the distribution of similarities between motifs found with LeMoNADe on the datasets which contained motifs (bottom).
Revised: We also show the distribution of similarities between motifs found with LeMoNADe on the datasets which contained motifs (bottom).

Original: The 95%-tile (corresponding to a 5% alpha level) of the BS distribution is displayed as vertical red line.
Revised: The 95%-tile (corresponding to a 5% alpha level) of the BS distribution is displayed as vertical red line.

Original: Up to a noise level of 70% the average of the similarities found on the datasets that contained motifs is much higher than the 95%-tile of the BS distribution.
Revised: Up to a noise level of 70% the average of the similarities found on the datasets that contained motifs is much higher than the 95%-tile of the BS distribution.

Original: D E XPERIMENTS AND RESULTS ON REAL DATA D. D ATA GENERATION
Revised: D E XPERIMENTS AND RESULTS ON REAL DATA D. D ATA GENERATION

Original: Organotypic hippocampal slice cultures were prepared from 7–9-day-old Wistar rats (Charles River
Revised: Organotypic hippocampal slice cultures were prepared from 7–9-day-old Wistar rats (Charles River

Original: Laboratories, Sulzfeld, Germany) as described by Kann et al.
Revised: Laboratories, Sulzfeld, Germany) as described by Kann et al.

Original: (2003) and Schneider et al.
Revised: (2003) and Schneider et al.

Original: Animals were taken care of and handled in accordance with the European directive 2010/63/EU and with consent of the animal welfare ofﬁcers.
Revised: Animals were taken care of and handled in accordance with the European directive 2010/63/EU and with consent of the animal welfare ofﬁcers at Heidelberg University (license, T96/15).

Original: Slices were infected with adeno-associated virus (AAV) obtained from Penn Vector Core (PA, USA) encoding GCaMP6f under the control of the CamKII promoter AAV5.CamKII.GCaMPf.
Revised: Slices were infected with adeno-associated virus (AAV) obtained from Penn Vector Core (PA, USA) encoding GCaMP6f under the control of the CamKII promoter AAV5.CamKII.GCaMPf.

Original: WPRE.SV40,
Revised: WPRE.SV40,

Original: Lot # V5392MI-S).
Revised: Lot # V5392MI-S).

Original: AAV transduction was achieved, under sterile conditions, by applying 0 .
Revised: AAV transduction was achieved, under sterile conditions, by applying 0 .

Original: 5 µl of the viral particles solution (qTiter: 1.55e13 GC/ml) on top of the slices.
Revised: 5 µl of the viral particles solution (qTiter: 1.55e13 GC/ml) on top of the slices.

Original: Slices were maintained on Biopore membranes (Millicell standing inserts; Merck Millipore, Schwalbach, Germany) between culture medium.
Revised: Slices were maintained on Biopore membranes (Millicell standing inserts; Merck Millipore, Schwalbach, Germany) between culture medium.

Original: The medium consisted of 50% minimal essential medium, 25% Hank’s balanced salt solution (Sigma-Aldrich, Taufkirchen, Germany), 25% horse serum (Life
Revised: The medium consisted of 50% minimal essential medium, 25% Hank’s balanced salt solution (Sigma-Aldrich, Taufkirchen, Germany), 25% horse serum (Life

Original: Technologies, Darmstadt, Germany), and 2 mM L-glutamine (Life Technologie) at pH 7 . 3 , stored in an incubator (Heracell; Thermoscientiﬁc, Dreieich, Germany) with humidiﬁed normal atmosphere (5% CO2, 36 . 5 ◦ C ).
Revised: Technologies, Darmstadt, Germany), and 2 mM L-glutamine (Life Technologie) at pH 7 . 3 , stored in an incubator (Heracell; Thermoscientiﬁc, Dreieich, Germany) with humidiﬁed normal atmosphere (5% CO2, 36 . 5 ◦ C ).

Original: The culture medium (1 ml) was replaced three times per week.
Revised: The culture medium (1 ml) was replaced three times per week.

Original: Artiﬁcial cerebrospinal ﬂuid used for imaging was composed of 129 mM NaCl, 3 mM KCl, 1.25 mM
Revised: Artiﬁcial cerebrospinal ﬂuid used for imaging was composed of 129 mM NaCl, 3 mM KCl, 1.25 mM

Original: NaH2PO4, 1.8 mM MgSO4, 1.6 mM CaCl2, 21 mM NaHCO3, and10 mM glucose (Sigma-Aldrich,
Revised: NaH2PO4, 1.8 mM MgSO4, 1.6 mM CaCl2, 21 mM NaHCO3, and10 mM glucose (Sigma-Aldrich,

Original: Taufkirchen, Germany).
Revised: Taufkirchen, Germany).

Original: The pH of the recording solution was 7.3 when it was saturated with the gas mixture (95% O2, 5% CO2).
Revised: The pH of the recording solution was 7.3 when it was saturated with the gas mixture (95% O2, 5% CO2).

Original: Recording temperature was 32 ± 1 ◦ C .
Revised: Recording temperature was 32 ± 1 ◦ C .

Original: Constant bath wash of 20 µM (dataset 1) and 10 µM (dataset 2) carbachol (Sigma-Aldrich) was performed to enhance neuronal activity and increase ﬁring probability during imaging (Müller et al., 1988). [Figure] [Figure]
Revised: Constant bath wash of 20 µM (dataset 1) and 10 µM (dataset 2) carbachol (Sigma-Aldrich) was performed to enhance neuronal activity and increase ﬁring probability during imaging (Müller et al

Original: [Figure]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: The fact that for both datasets in motif 0 some cells are displayed in red over multiple frames shows that these motifs contain temporal structure beyond mere spiking synchrony.
Revised: The fact that for both datasets in motif 0 some cells are displayed in red over multiple frames shows that these motifs contain temporal structure beyond mere spiking synchrony.

Original: Imaging of CA3 region of the hippocampus was performed on day 29 with 20x magniﬁcation (dataset 1) and on day 30 with 10x magniﬁcation (dataset 2) in vitro (23 days post viral infection) from slices maintained in submerged chamber of Olympus BX51WI microscope.
Revised: Imaging of CA3 region of the hippocampus was performed on day 29 with 20x magniﬁcation (dataset 1) and on day 30 with 10x magniﬁcation (dataset 2) in vitro (23 days post viral infection) from slices maintained in submerged chamber of Olympus BX51WI microscope.

Original: GCaMP6f was excited at 485 ± 10 nm .
Revised: GCaMP6f was excited at 485 ± 10 nm .

Original: Fluorescence images (emission at 521 ± 10 nm ) were recorded at 6 .
Revised: Fluorescence images (emission at 521 ± 10 nm ) were recorded at 6 .

Original: 4 Hz (dataset 1) and 4 Hz (dataset 2) using a CCD camera (ORCA-ER; Hamamatsu Photonics, Hamamatsu City,
Revised: 4 Hz (dataset 1) and 4 Hz (dataset 2) using a CCD camera (ORCA-ER; Hamamatsu Photonics, Hamamatsu City,

Original: Japan).
Revised: Japan).

Original: Before running the analysis we computed ∆ F/F for the datasets.
Revised: Before running the analysis we computed ∆ F/F for the datasets.

Original: In order to perform the computations more efﬁciently, we cropped the outer parts of the images containing no interesting neuronal activity and downsampled dataset 2 by a factor of 0 . 4 .
Revised: In order to perform the computations more efﬁciently, we cropped the outer parts of the images containing no interesting neuronal activity and downsampled dataset 2 by a factor of 0 . 4 .

Original: D. T EMPORAL STRUCTURE PLOTS
Revised: D. T EMPORAL STRUCTURE PLOTS

Original: In order to show that the motifs 0 found in the two real datasets contain temporal structure, we compare them to what the synchronous activity of the participating cells with modulated amplitude would look like.
Revised: In order to show that the motifs 0 found in the two real datasets contain temporal structure, we compare them to what the synchronous activity of the participating cells with modulated amplitude would look like.

Original: The synchronous ﬁring pattern was constructed as follows:
Revised: The synchronous ﬁring pattern was constructed as follows:

Original: First, for the motif M m with m = 1 , . . .
Revised: First, for the motif M m with m = 1 , . . .

Original: , M the maximum projection P m at each pixel p = 1 , . . .
Revised: , M the maximum projection P m at each pixel p = 1 , . . .

Original: , P · P (cid:48) over time was computed by 
Revised: , P · P (cid:48) over time was computed by P

Original: [Equation]
Revised: [Equation]

Original:  f = 1 , . . 
Revised: with f = 1 , . . .

Original: , F
Revised: , F

Original: [Equation]
Revised: [Equation]

Original:  and normalized
Revised: 19) and normalized

Original: [Equation]
Revised: [Equation]

Original: Finally, the synchronous ﬁring pattern S m for motif m is gained by multiplying this normalized maximum projection at each time frame f with the maximum intensity of motif m at that frame:
Revised: Finally, the synchronous ﬁring pattern S m for motif m is gained by multiplying this normalized maximum projection at each time frame f with the maximum intensity of motif m at that frame:

Original: [Equation]
Revised: [Equation]

Original: Figures 8 shows the difference between the found motif and the constructed synchronous ﬁring patterns for the motifs found on the two real datasets.
Revised: Figures 8 shows the difference between the found motif and the constructed synchronous ﬁring patterns for the motifs found on the two real datasets.

Original: D. C OMPARISON TO RESULTS OBTAINED WITH SCC
Revised: D. C OMPARISON TO RESULTS OBTAINED WITH SCC

Original: In order to show that LeMoNADe performs similar to SCC not only on synthetically generated data but also on real data, we ran both methods on real dataset 2.
Revised: In order to show that LeMoNADe performs similar to SCC not only on synthetically generated data but also on real data, we ran both methods on real dataset 2.

Original: A well trained neuroscientist manually extracted the individual cells and calcium traces from the original calcium imaging video.
Revised: A well trained neuroscientist manually extracted the individual cells and calcium traces from the original calcium imaging video.

Original: Figure 9a shows the result obtained with SCC on these traces.
Revised: Figure 9a shows the result obtained with SCC on these traces.

Original: In the same manner calcium traces were extraced from the motif found with LeMoNADe (see ﬁgure 9b).
Revised: In the same manner calcium traces were extraced from the motif found with LeMoNADe (see ﬁgure 9b).

Original: Both results in ﬁgure 9 are highly similar.
Revised: Both results in ﬁgure 9 are highly similar.

Original: E P ARAMETER SETTINGS
Revised: E P ARAMETER SETTINGS

Original: LeMoNADe is not more difﬁcult to apply than other motif detection methods for neuronal spike data.
Revised: LeMoNADe is not more difﬁcult to apply than other motif detection methods for neuronal spike data.

Original: In our experiments, for most of the parameters the default settings worked well on different datasets and only three parameters need to be adjusted: the maximum number of motifs M , the maximum 
Revised: In our experiments, for most of the parameters the default settings worked well on different datasets and only three parameters need to be adjusted: the maximum number of motifs M , the maximum [Figure] u

Original: [Figure]
Revised: [Figure]

Original:  n motif
Revised: motif [Figure] [Figure] n e u r o n motif

Original: [Figure] 
Revised: [Figure] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: motif length F , and one of the sparsity parameters (e.g. ˜ a or β KL ).
Revised: motif length F , and one of the sparsity parameters (e.g. ˜ a or β KL ).

Original: For SCC the user also has to specify three similar parameters.
Revised: For SCC the user also has to specify three similar parameters.

Original: In addition, SCC requires the previous extraction of a spike matrix which implies many additional parameters.
Revised: In addition, SCC requires the previous extraction of a spike matrix which implies many additional parameters.

Original: Table 3 shows the parameter settings used for the experiments shown in the paper.
Revised: Table 3 shows the parameter settings used for the experiments shown in the paper.

Original: E. O VER - AND UNDER - ESTIMATION OF THE MAXIMUM NUMBER OF MOTIFS M
Revised: E. O VER - AND UNDER - ESTIMATION OF THE MAXIMUM NUMBER OF MOTIFS M

Original: In order to show the effects of over- and underestimating the number of motifs, we ﬁrst use our synthetic data with existing ground truth and 3 true motifs and run LeMoNADe with underestimated ( M = 1 ), correct ( M = 3 ) and overestimated ( M = 5 ) number of expected motifs.
Revised: In order to show the effects of over- and underestimating the number of motifs, we ﬁrst use our synthetic data with existing ground truth and 3 true motifs and run LeMoNADe with underestimated ( M = 1 ), correct ( M = 3 ) and overestimated ( M = 5 ) number of expected motifs.

Original: Figure 10 shows the complete ground truth (ﬁgure 10a) and found motifs for the exemplary synthetic dataset discussed in the paper.
Revised: Figure 10 shows the complete ground truth (ﬁgure 10a) and found motifs for the exemplary synthetic dataset discussed in the paper.

Original: Besides the results for M = 3 (ﬁgure 10c) we also show the found motifs for M =(ﬁgure 10b) and M = 5 (ﬁgure 10d).
Revised: Besides the results for M = 3 (ﬁgure 10c) we also show the found motifs for M =(ﬁgure 10b) and M = 5 (ﬁgure 10d).

Original: If the number of motifs is underestimated ( M = 1 ) only one of the true motifs is captured.
Revised: If the number of motifs is underestimated ( M = 1 ) only one of the true motifs is captured.

Original: When the number of motifs is overestimated ( M = 5 ) the correct motifs are identiﬁed and the surplus ﬁlters are ﬁlled with (shifted) copies of the true motifs and background noise.
Revised: When the number of motifs is overestimated ( M = 5 ) the correct motifs are identiﬁed and the surplus ﬁlters are ﬁlled with (shifted) copies of the true motifs and background noise.

Original: We also investigated the inﬂuence of different numbers of motifs on the results on real datasets.
Revised: We also investigated the inﬂuence of different numbers of motifs on the results on real datasets.

Original: Figure 11 shows the found motifs on dataset 1 for the different numbers of motifs M = 1 , 2 , 3 , 5 .
Revised: Figure 11 shows the found motifs on dataset 1 for the different numbers of motifs M = 1 , 2 , 3 , 5 .

Original: When the number is limited (as for M = 1 ), the model is expected to learn those motifs ﬁrst which best explain the data.
Revised: When the number is limited (as for M = 1 ), the model is expected to learn those motifs ﬁrst which best explain the data.

Original: The motif shown in ﬁgure 11a also appears if M is increased.
Revised: The motif shown in ﬁgure 11a also appears if M is increased.

Original: This shows that this motif is highly present in the data.
Revised: This shows that this motif is highly present in the data.

Original: However, as long as only one ﬁlter is available the motif also contains a lot of background noise.
Revised: However, as long as only one ﬁlter is available the motif also contains a lot of background noise.

Original: The second ﬁlter in ﬁgure 11b contains a high luminosity artefact of the data.
Revised: The second ﬁlter in ﬁgure 11b contains a high luminosity artefact of the data.

Original: With its high luminosity and large spacial extent, it explains a lot of the dataset.
Revised: With its high luminosity and large spacial extent, it explains a lot of the dataset.

Original: However, it can easily be identiﬁed as no neuronal assembly.
Revised: However, it can easily be identiﬁed as no neuronal assembly.

Original: If the number of motifs is further increased to
Revised: If the number of motifs is further increased to

Original: M = 3 (see ﬁgure 11c), more background noise is captured in the additional ﬁlter and the motif becomes cleaner.
Revised: M = 3 (see ﬁgure 11c), more background noise is captured in the additional ﬁlter and the motif becomes cleaner.

Original: When the number of motifs is further increased to M = 5 , no new motifs appear and the surplus two ﬁlters seem to be ﬁlled up with parts of the structures which were already present in 11c.
Revised: When the number of motifs is further increased to M = 5 , no new motifs appear and the surplus two ﬁlters seem to be ﬁlled up with parts of the structures which were already present in 11c.

Original: Hence, when the correct number of motifs is unknown (as expected for real datasets) we recommend to slightly overestimate the expected number of motifs.
Revised: Hence, when the correct number of motifs is unknown (as expected for real datasets) we recommend to slightly overestimate the expected number of motifs.

Original: The result will capture the true motifs plus some copies of them.
Revised: The result will capture the true motifs plus some copies of them.

Original: In future work, a post-processing step as in Peter et al.
Revised: In future work, a post-processing step as in Peter et al.

Original: (2017) or a group sparsity regularization as in Bascol et al.
Revised: (2017) or a group sparsity regularization as in Bascol et al.

Original: (2016) and Mackevicius et al.
Revised: (2016) and Mackevicius et al.

Original: (2018) could be introduced to eliminate these additional copies automatically.
Revised: (2018) could be introduced to eliminate these additional copies automatically.

Original: Background noise could be easily identiﬁed as no motif by either looking at the motif videos or thresholding the found activations.
Revised: Background noise could be easily identiﬁed as no motif by either looking at the motif videos or thresholding the found activations.

Original: In future extends of the model we will study the effect of additional latent dimensions for background noise to automatically separate it from actual motifs.
Revised: In future extends of the model we will study the effect of additional latent dimensions for background noise to automatically separate it from actual motifs.

Original: E. O VER - AND UNDER - ESTIMATION OF THE MAXIMUM MOTIF LENGTH F
Revised: E. O VER - AND UNDER - ESTIMATION OF THE MAXIMUM MOTIF LENGTH F

Original: If the maximum motif length F is underestimated the found motifs are expected to just contain the part of the motif that reduces the reconstruction error most.
Revised: If the maximum motif length F is underestimated the found motifs are expected to just contain the part of the motif that reduces the reconstruction error most.

Original: Hence in most cases the most interesting parts of the motifs will be captured but details at either end of the motifs could be lost.
Revised: Hence in most cases the most interesting parts of the motifs will be captured but details at either end of the motifs could be lost.

Original: If the motif length is overestimated, the motifs can be captured completely but might be shifted in time.
Revised: If the motif length is overestimated, the motifs can be captured completely but might be shifted in time.

Original: This shift, however, will be compensated by the motif activations and hence has no negative effect on the results.
Revised: This shift, however, will be compensated by the motif activations and hence has no negative effect on the results.

Original: In our experiments we achieved good results with a generously chosen motif length.
Revised: In our experiments we achieved good results with a generously chosen motif length.

Original: For this reason we recommend to overestimate the motif length.
Revised: For this reason we recommend to overestimate the motif length.

Original: Figure 12 shows the found motifs on real dataset 1 with M = 3 and for the different motif lengths
Revised: Figure 12 shows the found motifs on real dataset 1 with M = 3 and for the different motif lengths

Original: F = 21 and F = 31 .
Revised: F = 21 and F = 31 .

Original: The results are highly similar.
Revised: The results are highly similar.

Original: In both cases, the interesting pattern (motif 0 in ﬁgure 12a and motif 1 in ﬁgure 12b, respectively) is captured.
Revised: In both cases, the interesting pattern (motif 0 in ﬁgure 12a and motif 1 in ﬁgure 12b, respectively) is captured.

Original: E. S PARSITY PARAMETER
Revised: E. S PARSITY PARAMETER

Original: The parameter ˜ a inﬂuences the sparsity of the found activations.
Revised: The parameter ˜ a inﬂuences the sparsity of the found activations.

Original: Smaller values of ˜ a will penalize activations harder and hence often result in cleaner and more meaningful motifs.
Revised: Smaller values of ˜ a will penalize activations harder and hence often result in cleaner and more meaningful motifs.

Original: However, if ˜ a is too small it will suppress the activations completely.
Revised: However, if ˜ a is too small it will suppress the activations completely.

Original: For this reason we recommend to perform for each new dataset experiments with different values of ˜ a .
Revised: For this reason we recommend to perform for each new dataset experiments with different values of ˜ a .

Original: Changing the value of β KL is another option to regulate the sparsity of the activations.
Revised: Changing the value of β KL is another option to regulate the sparsity of the activations.

Original: However, in our experiments we found that the default value of β KL = 0 .
Revised: However, in our experiments we found that the default value of β KL = 0 .

Original: 1 worked well for many different datasets and varying ˜ a was effective enough.
Revised: 1 worked well for many different datasets and varying ˜ a was effective enough.

Original: For the temperature parameters the default values λ 1 = 0 .
Revised: For the temperature parameters the default values λ 1 = 0 .

Original: 6 and λ 2 = 0 .
Revised: 6 and λ 2 = 0 .

Original: 5 worked well in most cases and changing them is usually not necessary.
Revised: 5 worked well in most cases and changing them is usually not necessary.

Original: In order to show the reaction of the method to the choice of ˜ a and β KL we performed multiple experiments on the real dataset 2 with different parameter settings.
Revised: In order to show the reaction of the method to the choice of ˜ a and β KL we performed multiple experiments on the real dataset 2 with different parameter settings.

Original: We ﬁxed all parameters as shown in table 3 except for ˜ a (ﬁgures 13 and 14) and β KL (ﬁgures 15 and 16).
Revised: We ﬁxed all parameters as shown in table 3 except for ˜ a (ﬁgures 13 and 14) and β KL (ﬁgures 15 and 16).

Original: When ˜ a is varied within one order of magnitude (see ﬁgure 13) the motifs look quite similar - except for temporal shifts of the motifs and shufﬂing of the order of the motifs.
Revised: When ˜ a is varied within one order of magnitude (see ﬁgure 13) the motifs look quite similar - except for temporal shifts of the motifs and shufﬂing of the order of the motifs.

Original: For smaller values of ˜ a surplus ﬁlters are ﬁlled with background noise (see ﬁgures 13a to 13d), whereas for a bit larger values of ˜ a the surplus ﬁlters are ﬁlled with copies of (parts of) the motif (see ﬁgures 13e to 13g).
Revised: For smaller values of ˜ a surplus ﬁlters are ﬁlled with background noise (see ﬁgures 13a to 13d), whereas for a bit larger values of ˜ a the surplus ﬁlters are ﬁlled with copies of (parts of) the motif (see ﬁgures 13e to 13g).

Original: Note that the motif which was also highlighted in the paper (ﬁgure 6d) appears in all results from ﬁgure 13b to 13g at least once.
Revised: Note that the motif which was also highlighted in the paper (ﬁgure 6d) appears in all results from ﬁgure 13b to 13g at least once.

Original: Only if ˜ a is changed by more than one order of magnitude the results become signiﬁcantly different and the motif is no longer detected (see ﬁgure 14).
Revised: Only if ˜ a is changed by more than one order of magnitude the results become signiﬁcantly different and the motif is no longer detected (see ﬁgure 14).

Original: This indicates that it is sufﬁcient to vary only the order of magnitude of ˜ a in order to ﬁnd a regime where motifs appear in the results and ﬁne tuning ˜ a is not necessary.
Revised: This indicates that it is sufﬁcient to vary only the order of magnitude of ˜ a in order to ﬁnd a regime where motifs appear in the results and ﬁne tuning ˜ a is not necessary.

Original: This strategy is also the recommended strategy to ﬁnd an appropriate sparsity parameter in SCC.
Revised: This strategy is also the recommended strategy to ﬁnd an appropriate sparsity parameter in SCC.

Original: A similar behavior can be observed when β KL is varied (see ﬁgure 15 for changes within an order of magnitude and ﬁgure 16 for larger changes).
Revised: A similar behavior can be observed when β KL is varied (see ﬁgure 15 for changes within an order of magnitude and ﬁgure 16 for larger changes).

Original: One can see similar effects as for the variation of ˜ a , but in the opposite direction: for smaller β KL surplus ﬁlters are rather ﬁlled with copies of the motif whereas for larger values of β KL the surplus ﬁlters are ﬁlled with background noise.
Revised: One can see similar effects as for the variation of ˜ a , but in the opposite direction: for smaller β KL surplus ﬁlters are rather ﬁlled with copies of the motif whereas for larger values of β KL the surplus ﬁlters are ﬁlled with background noise.

Original: This shows that it is usually sufﬁcient to only tune one of the two - either ˜ a or β KL - in order to achieve good results.
Revised: This shows that it is usually sufﬁcient to only tune one of the two - either ˜ a or β KL - in order to achieve good results

Original: [Table]
Revised: [Table]

Original: f r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m e( c ) F ound m o ti f s f
Revised: r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m e( c ) F ound m o ti f s f

Original: o r
Revised: o r

Original: M =motifmotifmotifmotiff
Revised: M =motifmotifmotifmotiff

Original: r a m e 0 motiff r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m e( d ) F ound m o ti f s f
Revised: r a m e 0 motiff r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m e( d ) F ound m o ti f s f

Original: o r
Revised: o r

Original: M =
Revised: M =

Original: F i gu r e 10 :
Revised: F i gu r e 10 :

Original: R e s u l t s fr o m t
Revised: R e s u l t s fr o m t

Original: h e e x e m p l a r y s yn t
Revised: h e e x e m p l a r y s yn t

Original: h e ti c
Revised: h e ti c

Original: d a t a s e t d i s c
Revised: d a t a s e t d i s c

Original: u
Revised: u

Original: ss e d i n t h
Revised: ss e d i n t h

Original: e p a p e r .
Revised: e p a p e r .

Original: ( a ) s ho
Revised: ( a ) s ho

Original: w s t
Revised: w s t

Original: h e t h r ee g r ound t r u t h m o ti f s .
Revised: h e t h r ee g r ound t r u t h m o ti f s .

Original: W e a l s o s ho w t
Revised: W e a l s o s ho w t

Original: h e r e s u lt
Revised: h e r e s u lt

Original: s o f ou r a n a l
Revised: s o f ou r a n a l

Original: y
Revised: y

Original: s
Revised: s

Original: i s w it h ﬁ x e d
Revised: i s w it h ﬁ x e d

Original: m
Revised: m

Original: o ti f l e ng t h ( F = 31 )
Revised: o ti f l e ng t h ( F = 31 )

Original: f o r t
Revised: f o r t

Original: h
Revised: h

Original: e
Revised: e

Original: d i ff e r e n t nu m b e r s o f m o ti f s
Revised: d i ff e r e n t nu m b e r s o f m o ti f s

Original: M =( b ) ,
Revised: M =( b ) ,

Original: M =( c ) a nd
Revised: M =( c ) a nd

Original: M =( d ) .
Revised: M =( d ) .

Original: f r a m e 
Revised: f r a m e motiff r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f

Original: [Table]
Revised: [Table]

Original: f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e 
Revised: m e [Table]

Original: [Table]
Revised: [Table]

Original: e
Revised: e

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e ( c ) F ound m o ti f s f
Revised: f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e ( c ) F ound m o ti f s f

Original: o r
Revised: o r

Original: M =motifmotifmotifmotiff
Revised: M =motifmotifmotifmotiff

Original: r a m e motiff r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e ( d ) F ound m o ti f s f o r
Revised: r a m e motiff r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e f r a m e ( d ) F ound m o ti f s f o r

Original: M =
Revised: M =

Original: F i gu r e 11 :
Revised: F i gu r e 11 :

Original: R e s u lt s fr o m d a t a s e tw
Revised: R e s u lt s fr o m d a t a s e tw

Original: it
Revised: it

Original: h ﬁ x e d m o ti f l e ng t h ( F = 21 )
Revised: h ﬁ x e d m o ti f l e ng t h ( F = 21 )

Original: f
Revised: f

Original: o r t
Revised: o r t

Original: h
Revised: h

Original: e
Revised: e

Original: d i ff e r e n t nu m b e r s o f m o ti f s ( a ) M = 1 , ( b )
Revised: d i ff e r e n t nu m b e r s o f m o ti f s ( a ) M = 1 , ( b )

Original: M = 2 , ( c )
Revised: M = 2 , ( c )

Original: M = 3 , a nd ( d )
Revised: M = 3 , a nd ( d )

Original: M = 5 .
Revised: M = 5 .

Original: motifmotiff r a m e 0 motiff r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m e( a ) F ound m o ti f s f
Revised: motifmotiff r a m e 0 motiff r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m e( a ) F ound m o ti f s f

Original: o r
Revised: o r

Original: F = motifmotiff r a m e 0 motiff r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r
Revised: F = motifmotiff r a m e 0 motiff r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r a m ef r

Original: a m e( b ) F ound m o ti f s f
Revised: a m e( b ) F ound m o ti f s f

Original: o r
Revised: o r

Original: F =
Revised: F =

Original: F i gu r e 12 :
Revised: F i gu r e 12 :

Original: R e s u lt s fr o m d a t a s e tw
Revised: R e s u lt s fr o m d a t a s e tw

Original: it
Revised: it

Original: h ﬁ x e
Revised: h ﬁ x e

Original: d nu m b e r o f m o ti f s (
Revised: d nu m b e r o f m o ti f s (

Original: M = 3 ) f
Revised: M = 3 ) f

Original: o r t
Revised: o r t

Original: h
Revised: h

Original: e
Revised: e

Original: d i ff e r e n t m o ti f l e
Revised: d i ff e r e n t m o ti f l e

Original: ng t h s ( a )
Revised: ng t h s ( a )

Original: F = 21 a nd ( b )
Revised: F = 21 a nd ( b )

Original: F = 31 .
Revised: F = 31 .

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: F M OTIF VIDEOS
Revised: F M OTIF VIDEOS

Original: In order to give the reader a better impression of what the used data and the motifs extracted as short video sequences would look like, we provide a few video ﬁles containing extracted motifs, m o t i fm o t
Revised: In order to give the reader a better impression of what the used data and the motifs extracted as short video sequences would look like, we provide a few video ﬁles containing extracted motifs, m o t i fm o t

Original: i fm o t
Revised: i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: The reconstructed videos are gained by convolving the found motifs with the corresponding found activations.
Revised: The reconstructed videos are gained by convolving the found motifs with the corresponding found activations.

Original: The videos are provided either in TIFF or MP4 format.
Revised: The videos are provided either in TIFF or MP4 format.

Original: Table 4 shows the names of the ﬁles together with short descriptions what each video shows.
Revised: Table 4 shows the names of the ﬁles together with short descriptions what each video shows.

Original: The videos corresponding to the synthetic dataset were generated with a frame rate of 30 fps and those corresponding to the real dataset with 10 fps .
Revised: The videos corresponding to the synthetic dataset were generated with a frame rate of 30 fps and those corresponding to the real dataset with 10 fps .

Original: m
Revised: m

Original: o t i fm o t 
Revised: o t i fm o t i fm o t i f

Revised: [Figure]

Original:  i fm o t 
Revised: m o t i fm o t i fm o t i f

Revised: [Figure]

Revised: m o t

Revised: i fm o t

Revised: i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t 
Revised: i fm o t i fm o t i f

Revised: [Figure]

Original:  i fm o t
Revised: m o t i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: m o t

Original: i fm o t i fm o t
Revised: i fm o t i fm o t

Original: i f
Revised: i f

Original: [Figure]
Revised: [Figure]

Original: m o t
Revised: [Figure]

Original: i fm o t i fm o t i f
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: m o t i fm o t i fm o t i f
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: m o t i fm o t i fm o t i f
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Table] 
Revised: [Table] [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: video

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: motif [Table] [Table] motif

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Table] motif [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: video [Table] [Table] [Table]

Revised: 21 synth

Original: [Table]
Revised: [Table]

Revised: example

Original: [Figure]
Revised: [Table]

Revised: synth

Original: [Table]
Revised: [Table]

Revised: example

Original: [Figure]
Revised: [Table]

Revised: 21 synth

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: example [Table] example [Table]

Revised: 21 synth

Original: [Table]
Revised: [Table]

Revised: example

Original: [Figure]
Revised: [Table]

Revised: 21 synth

Original: [Table]
Revised: [Table]

Revised: example

Original: [Table]
Revised: [Table]

Revised: video

Original: [Table]
Revised: [Table]

Revised: 21 synth

Original: [Table]
Revised: [Table]

Revised: example

Original: [Table]
Revised: [Table]

Revised: 21 synth

Original: [Table]
Revised: [Table]

Revised: 21 synth

Original: [Table]
Revised: [Table]