S YNTHESIZING REALISTIC NEURAL POPULATION ACTIVITY PATTERNS USING G ENERATIVE A DVERSARIAL N ETWORKS
Corresponding author ∗ Equal contribution molano stefano panzeri iit
A BSTRACT
The ability to synthesize realistic patterns of neural activity is crucial for studying neural information processing.
Here we used the Generative Adversarial Networks (GANs) framework to simulate the concerted activity of a population of neurons.
We adapted the Wasserstein-GAN variant to facilitate the generation of unconstrained neural population activity patterns while still beneﬁting from parameter sharing in the temporal domain.
We demonstrate that our proposed GAN, which we termed Spike-GAN, generates spike trains that match accurately the ﬁrst- and second-order statistics of datasets of tens of neurons and also approximates well their higher-order statistics.
We applied Spike-GAN to a real dataset recorded from salamander retina and showed that it performs as well as state-ofthe-art approaches based on the maximum entropy and the dichotomized Gaussian frameworks.
Importantly, Spike-GAN does not require to specify a priori the statistics to be matched by the model, and so constitutes a more ﬂexible method than these alternative approaches.
Finally, we show how to exploit a trained SpikeGAN to construct ’importance maps’ to detect the most relevant statistical structures present in a spike train.
Spike-GAN provides a powerful, easy-to-use technique for generating realistic spiking neural activity and for describing the most relevant features of the large-scale neural population recordings studied in modern systems neuroscience.
I NTRODUCTION
Understanding how to generate synthetic spike trains simulating the activity of a population of neurons is crucial for systems neuroscience.
In computational neuroscience, important uses of faithfully generated spike trains include creating biologically consistent inputs needed for the simulation of realistic neural networks, generating large datasets to be used for the development and validation of new spike train analysis techniques, and estimating the probabilities of neural responses in order to extrapolate the information coding capacity of neurons beyond what can be computed from the neural data obtained experimentally (Ince et al., 2013; Moreno-Bote et al., 2014).
In experimental systems neuroscience, the ability to develop models that produce realistic neural population patterns and that identify the key sets of features in these patterns is fundamental to disentangling the encoding strategies used by neurons for sensation or behavior (Panzeri et al., 2017) and to design closed-loop experiments (Kim et al., 2017) in which synthetic patterns, representing salient features of neural information, are fed to systems of electrical micro-stimulation (Tehovnik et al., 2006) or patterned light optogenetics (Panzeri et al., 2017; Bovetti & Fellin, 2015) for naturalistic intervention on neural circuits.
One successful way to generate realistic spike trains is that of using a bottom-up approach, focusing explicitly on replicating selected low-level aspects of spike trains statistics.
Popular methods include renewal processes (Stein (1965); Gerstner & Kistler (2002)), latent variable models (Macke et al., 2009; Lyamzin et al., 2010) and maximum entropy approaches (Tang et al., 2008; Schneidman et al., 2006; Savin & Tkaˇcik, 2017), which typically model the spiking activity under the assumption that only ﬁrst and second-order correlations play a relevant role in neural coding (but see Cayco-Gajic et al. (2015); K¨oster et al.
; Ohiorhenuan et al.
Other methods model spike train responses assuming linear stimulus selectivity and generating single trial spike trains using simple models of input-output neural nonlinearities and neural noise (Keat et al., 2001; Pillow et al., 2008;
Lawhern et al., 2010).
These methods have had a considerable success in modeling the activity of populations of neurons in response to sensory stimuli (Pillow et al., 2008).
Nevertheless, these models are not completely general and may fail to faithfully represent spike trains in many situations.
This is because neural variability changes wildly across different cortical areas (Maimon & Assad, 2006) due to the fact that responses, especially in higher-order areas and in behaving animals, have complex non-linear tuning to many parameters and are affected by many behavioral variables (e.g. the level of attention (Fries et al., 2001)).
An alternative approach is to apply deep-learning methods to model neural activity in response to a given set of stimuli using supervised learning techniques (McIntosh et al., 2016).
The potential advantage of this type of approach is that it does not require to explicitly specify any aspect of the spike train statistics.
However, applications of deep networks to generate faithful spike patterns have been rare.
Here, we explore the applicability of the Generative Adversarial Networks (GANs) framework (Goodfellow et al., 2014) to this problem.
Three aspects of GANs make this technique a good candidate to model neural activity.
First, GANs are an unsupervised learning technique and therefore do not need labeled data (although they can make use of labels (Odena et al., 2016b;
Chen et al., 2016)).
This greatly increases the amount of neural data available to train them.
Second, recently proposed modiﬁcations of the original GANs make them good at ﬁtting distributions presenting multiple modes (Arjovsky et al., 2017; Gulrajani et al., 2017).
This is an aspect that is crucial for neural data because the presentation of even a single stimulus can elicit very different spatio-temporal patterns of population activity (Churchland et al., 2007; Morcos & Harvey, 2016).
We thus need a method that generates sharp realistic samples instead of producing samples that are a compromise between two modes (which is typical, for instance, of methods seeking to minimize the mean squared error between the desired output and the model’s prediction (Goodfellow, 2016;
Lotter et al., 2016)).
Finally, using as their main building block deep neural networks, GANs inherit the capacity of scaling up to large amounts of data and therefore constitute a good candidate to model the ever growing datasets provided by experimental methods like chronic multi-electrode and optical recording techniques.
In the present work we extend the GAN framework to synthesize realistic neural activity.
We adapt the recently proposed Wasserstein-GAN (WGAN) (Arjovsky et al., 2017) which has been proven to stabilize training, by modifying the network architecture to model invariance in the temporal dimension while keeping dense connectivity across the modeled neurons.
We show that the proposed
GAN, which we called Spike-GAN, is able to produce highly realistic spike trains matching the ﬁrst and second-order statistics of a population of neurons.
We further demonstrate the applicability of Spike-GAN by applying it to a real dataset recorded from the salamander retina (Marre et al., 2014) and comparing the activity patterns the model generates to those obtained with a maximum entropy model (Tkaˇcik et al., 2014) and with a dichotomized Gaussian method (Lyamzin et al., 2010).
Finally, we describe a new procedure to detect, in a given activity pattern, those spikes participating in a speciﬁc feature characteristic of the probability distribution underlying the training dataset.
M ETHODS 2. N ETWORK ARCHITECTURE
We adapted the Generative Adversarial Networks described by Goodfellow et al.
(2014) to produce samples that simulate the spiking activity of a population of N neurons as binary vectors of length
T (spike trains; Fig. S3).
In their original form, GANs proved to be difﬁcult to train, prompting several subsequent studies that focused on making them more stable (Radford et al., 2015; Chintala et al., 2016).
In the present work we used the Wasserstein-GAN variant described by Arjovsky et al.
Wasserstein-GANs (WGAN) minimize the Earth-Mover (or Wasserstein-1) distance
(EM) between the original distribution P data and the distribution deﬁned by the generator, P G .
Arjovsky et al.
(2017) showed that the EM distance has desirable properties in terms of continuity and differentiability that ensure that the loss function provides a meaningful gradient at all stages of training, which boosts considerably its stability.
A further improvement was later introduced by
Gulrajani et al.
(2017), who provided an alternative procedure to ensure that the critic is Lipschitz (via gradient penalization), which is required in the WGAN framework.
Here we adapted the WGAN-GP architecture (Gulrajani et al., 2017) to simulate realistic neural population activity patterns.
Our samples are matrices of size N × T , where N is the number of neurons and T the number of time bins, each bin usually corresponding to a few milliseconds (Fig. S3).
Importantly, while samples present a high degree of invariance along the time dimension, they are usually not spatially structured (i.e. across neurons) and thus we cannot expect any invariance along the dimension spanning the different neurons.
For this reason, in order to take advantage of the temporal invariance while being maximally agnostic about the neural correlation structure underlying the population activity, we modiﬁed a standard 1D-DCGAN (1 dimensional deep convolutional
GAN) architecture (Radford et al., 2015) by transposing the samples so as to make the spatial dimension correspond to the channel dimension (Fig. 1).
Therefore our proposed GAN can be seen as performing a semi -convolution, where the spatial dimension is densely connected while weights are shared across the temporal dimension thus improving training, efﬁciency and the interpretability of the trained networks.
The main modiﬁcations we have introduced to the WGAN-GP are:
The responses of different neurons are fed into different channels.
Following Chintala et al.
we made all units LeakyReLU
(the slope of the leak was set to 0.2) except for the last layer of the generator where we used sigmoid units.
The critic consists of two 1D convolutional layers with 256 and 512 features, respectively, followed by a linear layer (Fig. 1).
The generator samples from a 128-dimension uniform distribution and its architecture is the mirror image of that of the critic.
To avoid the checkerboard issue described by Odena et al.
(2016a), we divided all generator’s fractional-strided convolutions (i
e. deconvolutions) into two separate steps: upsampling and convolving.
The upsampling step is done using a nearest neighbor procedure, as suggested by Odena et al.
(2016a).
We called the network described above Spike-GAN.
As in Arjovsky et al.
(2017), Spike-GAN was trained with mini-batch stochastic gradient descent (we used a mini-batch size of 64).
All weights were initialized from a zero-centered normal distribution with standard deviation 0.02.
We used the Adam optimizer (Kingma & Ba, 2014) with learning rate = 0.0001 and hyperparameters β 1 = 0 and β 2 = 0 .
The parameter λ , used for gradient penalization, was set to 10.
The critic was updated 5 times for each generator update.
All code and hyperparameters may be found at https://github.com/manuelmolano/Spike-GAN.
2. S PIKE TRAIN ANALYSIS
To compare the statistics of the generated samples to the ones contained in the ground truth dataset, we ﬁrst discretized the continuously-valued samples produced by the generator and then, for each bin with activation h , we drew the ﬁnal value from a Bernoulli distribution with probability h .
Note that the last layer of the generator contains a sigmoid function and thus the h values can be interpreted as probabilities.
We assessed the performance of the model by measuring several spike train statistics commonly used in neuroscience: 1) Average number of spikes (spike-count) per neuron.
Average time course, which corresponds to the probability of ﬁring in each bin, divided by the bin duration (measured in seconds).
Covariance between pairs of neurons.
4) Lag-covariance between pairs of neurons: for each pair of neurons, we shift the activity of one of the neurons by one bin and compute the covariance between the resulting activities.
This quantity thus indicates how strongly the activity of one of the neurons is related to the future activity of the other neuron.
5) Distribution of synchrony (or k-statistic), which corresponds to the probability P N ( k ) that k
out of the N neurons spike at the same time.
6) Spike autocorrelogram, computed by counting, for each spike, the number of
Critic's Architecture n e u r o n s (
N ) t i m e b i n s ( T )
T /
T /f e a t u r e s ( 256 ) f e a t u r e s ( 512 ) f
il t e r N x
C o n v o l u t i o n ( s t r i d e 2 ) + L e a ky R e L U
F l a tt e n
L i n e a r p r o j e c t i o n
C o n v o l u t i o n ( s t r i d e 2 ) + L e a ky R e L U
C r i t i c ' s o u t p u t spikes preceding and following the given spike in a predeﬁned time window.
The obtained trace is normalized to the peak (which is by construction at 0 ms) and the peak is then zeroed in order to help comparisons.
R ESULTS 3. F ITTING THE STATISTICS OF SIMULATED SPIKE TRAINS
We ﬁrst tested Spike-GAN with samples coming from the simulated activity of a population of 16 neurons whose ﬁring probability followed a uniform distribution across the whole duration (T=128 ms) of the samples (bin size=1 ms, average ﬁring rate around 100 Hz, Fig.
2D).
In order to test whether Spike-GAN can approximate second-order statistics, the neurons’ activities present two extra features that are commonly found in neural recordings.
First, using the method described in Mikula & Niebur (2003), we introduced correlations between randomly selected pairs of neurons (8 pairs of correlated neurons; correlation coefﬁcient values around 0.3).
Second, we imposed a common form of temporal correlations arising from neuronal biophysics ( refractory period ): following an action potential, a neuron typically remains silent for a few milliseconds before it is able to spike again.
This phenomenon has a clear effect on the spike autocorrelogram that shows a pronounced drop in the number of spikes present at less than 2 ms (see Fig. 2E).
We trained Spike-GAN on 8192 samples for 500000 iterations (Fig.
S4 shows the critic’s loss function across training).
A representative sample produced by a trained Spike-GAN together with the resulting patterns (after binarizing the samples, see Section 2.2) is shown in Fig.
2A. Note that the sample (black traces) is mostly binary, with only a small fraction of bins having intermediate values between 0 and 1.
We evaluated the performance of Spike-GAN by measuring several spike train statistics commonly used in neuroscience (see Section 2.2).
For comparison, we also trained a generative adversarial network in which both the generator and the critic are a 4-layer multi-layer perceptron (MLP) and the number of units per layer is adjusted so both models present comparable numbers of trainable variables ( units per layer which results in ≈ 3 .
5 M trainable variables).
As Fig.
2 shows, while both models ﬁt fairly well the ﬁrst three statistics (mean spike-count, covariances and k-statistics), the SpikeGAN’s approximation of the features involving time (average time course, autocorrelogram and lag-covariance) is considerably better than that of the MLP GAN.
This is most likely due to the weight sharing performed by Spike-GAN along the temporal dimension, that allows it to easily learn temporally invariant features.
In Supp.
Section A.1 we further show that Spike-GAN is not only memorizing the samples present in the training dataset
but it is able to effectively mimic their underlying distribution.
3. C OMPARING TO STATE - OF - THE - ART METHODS
We next tested the Spike-GAN model on real recordings coming from the retinal ganglion cells (RGCs) of the salamander retina (Marre et al., 2014; Tkaˇcik et al., 2014).
The dataset contains the response of 160 RGCs to natural stimuli (297 repetitions of a 19-second movie clip of swimming ﬁsh and water plants in a ﬁsh tank) discretized into bins of 20 ms.
We randomly selected 50 neurons out of the total 160 and partitioned their activity into non-overlapping samples of 640 ms (32 time bins) which yielded a total of 8817 training samples (using overlapping samples and thus increasing their number did not improve the results shown below).
We obtained almost identical results for a different set of 50 randomly selected neurons (data not shown).
In order to provide a comparison between Spike-GAN and existing state-of-the-art methods, we ﬁt the same dataset with a maximum entropy approach developed by Tkaˇcik et al.
(2014), the so-called k-pairwise model, and a dichotomized Gaussian method proposed by Lyamzin et al.
Brieﬂy, maximum entropy (MaxEnt) models provide a way of ﬁtting a predeﬁned set of statistics characterizing a probability distribution while being maximally agnostic about any other aspect of such distribution, i.e. maximizing the entropy of the probability distribution given the constraints in the statistics (Press´e et al., 2013).
In neuroscience applications, the most common approach has been to design MaxEnt models ﬁtting the ﬁrst and second-order statistics, i.e. the average ﬁring rates and pairwise correlations between neurons (Tang et al., 2008; Schneidman et al., 2006; Shlens et al., 2006).
The k-pairwise model extends this approach to further constrain the activity of the neural population by ﬁtting the k-statistics of the dataset of interest, which provides a measure of the neural population synchrony (see Section 2.2).
Dichotomized Gaussian (DG) methods, on the other hand, model the neural activity by thresholding a correlated multivariate normal distribution with mean and covariance chosen such that the generated samples have the desired ﬁrst- and second-order statistics.
The method developed by Lyamzin et al.
(2010) is an extension of previous approaches (see e.g. Macke et al.
in which signal and noise correlations are modeled separately.
Importantly, unlike the k-pairwise model (and most MaxEnt models (Savin & Tkaˇcik, 2017)), the DG model can ﬁt temporal correlations.
We ﬁrst checked for signs of overﬁtting by plotting, for each method, randomly selected generated samples together with their closest sample (in terms of L1 distance) in the training dataset.
Although the generator in a GAN never ’sees’ the training dataset directly but instead obtains information about the dataset only through the critic, it is still possible that the generator obtains enough information about the real samples to memorize them.
Fig.
S5 shows that this is not the case, with the closest samples in the training dataset being very different from the generated ones.
As shown in Fig.
3, all methods provide a good approximation of the average ﬁring rate, the covariance and the k-statistics, but the ﬁt performed by the MaxEnt (green dots) and the DG (blue pluses) models is somewhat tighter than that produced by Spike-GAN (red dots).
This is not surprising, as these are the aspects of the population activity distribution that these models are speciﬁcally designed to ﬁt.
By contrast, Spike-GAN does remarkably well without any need for these statistical structures to be manually speciﬁed as features of the model.
As mentioned above, the k-pairwise model does not take into account the temporal dynamics of the population and therefore ignores well-known neural features that are very likely to play a relevant role in the processing of incoming information (e.g. refractory period, burst or lagged crosscorrelation between pairs of neurons).
Fig.
3 shows that both Spike-GAN and the DG model approximate well the ground truth autocorrelogram and lag-covariances while the k-pairwise model, as expected, entirely fails to do so.
Importantly, while its performance in terms of reproducing positive correlations is remarkable, the
DG method struggles to approximate the statistics of neural activity associated with negative correlations (Lyamzin et al., 2010).
Fig.
S6 shows how the k-pairwise and the DG methods ﬁt the dataset described in Fig.
As can be seen, the DG model, while matching perfectly the (positive) correlations between neurons, fails to approximate the negative correlations present in the autocorrelogram that are caused by the refractory period (Fig.
S6E).
The above results demonstrate that Spike-GAN generates samples comparable to those produced by state-of-the-art methods without the need of deﬁning a priori which statistical structures constitute important features of the probability distribution underlying the modeled dataset.
3. U SING THE TRAINED CRITIC TO INFER RELEVANT NEURAL FEATURES
We then investigated what a trained critic can tell us about the population activity patterns that compose the original dataset.
In order to do so, we designed an alternative dataset in which neural samples contain stereotyped activation patterns each involving a small set of neurons (Fig. 4A).
This type of activation patterns, also called packets, have been found in different brain areas and have been suggested to be fundamental for cortical coding, forming the basic symbols used by populations of neurons to process and communicate information about incoming stimuli (Luczak et al., 2015).
Thus, besides being a good test for the capability of Spike-GAN to approximate more intricate statistical structures, analyzing simulated samples presenting packets constitutes an excellent way of demonstrating the applicability of the model to a highly relevant topic in neuroscience.
We trained
Spike-GAN on a dataset composed of neural patterns of 32 neurons by 64 ms that present four different packets involving non-overlapping sets of 8 neurons each (Fig. 4A).
Importantly, only few neurons out of all the recorded ones typically participate in a given packet and, moreover, neurons are usually not sorted by the packet to which they belong.
Therefore, real neural population activity is extremely difﬁcult to interpret and packets are cluttered by many other ’noisy’ spikes (Fig. 4B).
In order to assess the applicability of Spike-GAN to real neuroscience experiments, we trained it on these type of realistic patterns of activity (see caption of Fig.
4 for more details on the simulated dataset and the training).
Visual inspection of the ﬁlters learned by the ﬁrst layer of the critic suggests that Spike-GAN is able to learn the particular structure of the packets described above: many of the ﬁlters display spatial distributions that are ideally suited for packet detection (Fig.
S7; note that ﬁlters have been sorted in the neurons’ dimension to help visualization).
Recently, Zeiler & Fergus (2014) developed a procedure to investigate which aspects of a given sample are most relevant for a neural network.
They proposed to systematically alter different parts of the input and evaluate the change each alteration produces in the output of different layers of the network.
Here we have adapted this idea to investigate which are the most relevant features of a given neural activity pattern.
We ﬁrst compute the output produced by the critic for a real sample.
Then, for a given neuron and a given temporal window of several milliseconds, we shufﬂe across time the spikes emitted by the neuron during that period of time and compute the output of the critic when using as input the altered sample.
The absolute difference between the two outputs gives us an idea of how important is the structure of the spike train we have disrupted.
We can then proceed in the same fashion for all neurons and for several time windows and obtain a map of the importance of each particular spike train emitted by each neuron (importance maps, see Fig.
4C, heatmaps).
To highlight the usefulness of the procedure explained above, we produced a separate dataset in which the same population of neurons encodes the information about a particular stimulus by emitting one of the packet types shown in Fig.
4A around 16 ms after the stimulus presentation 1 .
Fig.
4C (gray scale panels) shows 5 representative example patterns (see also Fig. S8).
The packets are highlighted for visualization, but it is clear that patterns containing packets are almost indistinguishable from those without them.
Noticeably, the importance maps (heatmaps) are able to pinpoint the spikes belonging to a packet (note that this does not require re-training of Spike-GAN).
Further, by averaging the importance maps across time and space, we can obtain unambiguous results regarding the relevance of each neuron and time period (Fig.
4D-E; in Fig.
4E the neurons presenting higher importance values are those participating in the packet).
The importance-map analysis thus constitutes a very useful procedure to detect the most relevant aspects of a given neural population activity pattern.
In Fig.
S2 we describe a potential application of the importance maps to the study of how a population of neurons encode the information about a given set of stimuli.
D ISCUSSION
We explored the application of the Generative Adversarial Networks framework (Goodfellow et al., 2014) to synthesize neural responses that approximate the statistics of the activity patterns of a population of neurons.
For this purpose, we put forward Spike-GAN, by adapting the WGAN variant proposed by Arjovsky et al.
(2017) to allow sharing weights across time while maintaining a densely connected structure across neurons.
We found that our method reproduced to an excellent approximation the spatio-temporal statistics of neural activity on which it was trained.
Importantly, it does so without the need for these statistics to be handcrafted in advance, which avoids making a priori assumptions about which features of the external world make neurons ﬁre.
Recently, Pandarinath et al.
(2017) have proposed a deep learning method, LFADS (Latent Factor
Analysis via Dynamical Systems), to model the activity of a population of neurons using a variational autoencoder (in which the encoder and decoder are recurrent neural networks).
LFADS allows inferring the trial-by-trial population dynamics underlying the modeled spike train patterns and thus can be seen as a complementary method to Spike-GAN, which does not explicitly provide the latent factors governing the response of the neurons.
Regarding the application of the GANs framework to the ﬁeld of neuroscience, Arakaki et al.
(2017) proposed a GAN-based approach for ﬁtting network models to experimental data consisting of a set of tuning curves extracted from a population of neurons.
However, to the best of our knowledge our work is the ﬁrst to use GANs to directly produce realistic neural patterns simulating the activity of populations of tenths of neurons.
Building on the work by Zeiler & Fergus (2014), we showed how to use Spike-GAN to visualize the particular features that characterize the training dataset.
Speciﬁcally, Spike-GAN can be used to obtain importance maps that highlight the spikes that participate in generating activity motifs that are most salient in the spike trains.
This can be useful for unsupervised identiﬁcation of highly salient low-dimensional representations of neural activity, which can then be used to describe and interpret experimental results and discover the key units of neural information used for functions such as sensation and behavior.
A further and promising application of importance maps is that of designing realistic patterns of stimulation that can be used to perturb populations of neurons using electrical or optical neural stimulation techniques (Panzeri et al., 2017; Tehovnik et al., 2006; Emiliani et al., 2015).
The ability of
Spike-GAN to generate realistic neural activity including its temporal dynamics and to identify its most salient features suggests that it may become a very relevant tool to design perturbations.
In
Fig.
S2 we provide a more detailed description of a potential application of Spike-GAN, in which importance maps may allow inferring the set of neurons participating in the encoding of the information about a given set of stimuli (Fig.
S2F)
and the spatio-temporal structure of the packets elicited by each stimulus (Fig. S2E).
We have compared Spike-GAN with two alternative methods based on the maximum entropy and the dichotomized Gaussian frameworks.
These methods offer the possibility of computing the sample probabilities (MaxEnt model) and separately specifying the signal and noise correlations present in the generated samples (DG model).
Spike-GAN does not have these features; nevertheless, it does have important advantages over the mentioned methods.
First, Spike-GAN is more ﬂexible than the MaxEnt and DG models, being able to ﬁt any type of spatio-temporal structure present in the data.
Further, it does not require making a priori assumptions about which statistical properties of a dataset are relevant and thus need to be matched.
Finally, Spike-GAN is based on the deep neural network framework, and is therefore able to directly beneﬁt from the engineering advances emerging in this rapidly-growing ﬁeld.
Conceivably, this will enable Spike-GAN, or methods derived from it, to make in the future better and better use of the datasets of ever increasing size that are produced by the experimental neuroscience community
A CKNOWLEDGMENTS
This work has received funding from the European Unions Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 699829 (”ETIC”) and under the Marie Sklodowska-Curie grant agreement No 659227 (STOMMAC).
EP thanks Vittal Premachandran for discussions at the Brains, Minds and Machines summer course.
A A PPENDIX A. G ENERATING NEW , REALISTIC PATTERNS OF NEURAL ACTIVITY
In this section, we investigated how well Spike-GAN ﬁts the whole probability density function from which the population activity patterns present in the training dataset are drawn, following an approach inspired by Macke et al.
We started by producing a ground truth dataset ( 2 · 10 6 samples) for a small-size problem (2 neurons x 12 time bins) so as to reduce the dimensionality of the samples and obtain a good approximation of the underlying probability density function.
We will call the probabilities computed from the ground truth dataset numerical probabilities .
We then obtained a second dataset ( 2 · 10 6 samples) drawn from the same probability distribution as the ground truth dataset, which we will call surrogate dataset.
This dataset will provide us with a reference for the results we obtain when comparing the distribution generated by Spike-GAN with the original ground truth distribution.
A third small dataset ( training dataset, 8192 samples) coming from the same probability distribution as the ground truth and surrogate datasets, was used to train Spike-GAN.
Finally, a 2 · 10 6 -sample dataset ( generated dataset) was obtained from the trained Spike-GAN.
The distributions corresponding to the three different datasets presented very similar entropies: the ground truth and surrogate datasets had both 14.6 bits while the generated dataset had 14.3 bits.
Thus, the dataset generated by Spike-GAN was not affected by any evident mode collapse.
We then plotted the sample probabilities with respect to both the generated and the surrogate dataset against the numerical probabilities .
By comparing the densities in Fig.
S1, we deduce that the surrogate probability distribution deviates from the ground truth distribution (the identity line) in the same way as the generated distribution does.
Hence, this deviation can be attributed to ﬁnite sampling effects rather than poor performance of Spike-GAN.
Importantly, the percentage of samples generated by Spike-GAN that were originally present in the training dataset was 44% ( 45% for the surrogate dataset).
This implies that 56% percent of the samples produced by Spike-GAN were generated de novo .
Finally, the percentage of generated samples that had numerical probability equal to 0 was 3 .
2% , which is comparable to the 3 . 8% of samples in the surrogate dataset that also had numerical probability 0.
Taken together, the above results strongly suggest that Spike-GAN has learned the probability distribution underlying the training dataset.
A. U SING IMPORTANCE MAPS IN A REAL EXPERIMENT
When recording large-scale spike trains it is difﬁcult to make hypotheses about what are the key features of population activity that allow the animal to discriminate between the stimuli (for example, the patterns of ﬁring rates or the pattern of response latencies of speciﬁc subsets of neurons).
One possible use of Spike-GAN is to interpret the features of neural activity that are prominent in the importance maps as possible candidates for being units of information relevant for behavioral discrimination.
To illustrate the approach, we simulated a hypothetical experiment (Fig.
S2A) in which we consider
N repetitions of a behavioural task, where a mouse has to discriminate two different stimuli (vertical/horizontal stripes).
For each repetition of the task, we assume to record several patterns of neural activity such as those in Fig.
4 (main paper).
By means of two-photon calcium imaging the activity of a population of V1 neurons in the visual cortex of the mouse is recorded in response to the two stimuli, which are associated with two distinct behavioral outcomes (e.g. drink from a left/right reward port (Alemi-Neissi et al., 2013)).
The mouse is assumed to have been previously trained on the task.
[Note that one possible difﬁculty in applying Spike-GAN to calcium imaging is that, unlike spikes directly extracted from electrophysiology experiments, two-photon imaging signals are not binary-valued.
Nevertheless, to a ﬁrst approximation, calcium signals can be binarized, and this has been shown not to constitute an obstacle to the study of sensory encoding or decision-making in certain preparations (Runyan et al., 2017).]
Area V1 is known to encode information about the stimulus orientation and therefore the two stimuli will evoke two distinct activity patterns (Fig. S2B).
However, these patterns are usually difﬁcult to identify due to background activity and the lack of information about the ground truth correlation structure of the recorded population of neurons.
Fig.
S2C shows the actual ’recorded’ population responses.
As described in Section 3.3, Spike-GAN can then be trained on the samples shown in Fig.
S2C in order to compute the importance map associated to each sample (Fig.
S2D).
We can then approximate the structure of the original packets by thresholding and aligning (to the ﬁrst spike above threshold) each importance map and averaging all aligned maps for each stimulus.
Importantly, the approximated packets (gray maps) and the original ones (green pluses) are highly correlated (Fig.
S2E; r ≥ 0 .
8 , yellow values) and the distribution of importance values corresponding to bins participating in a packet (those marked with a green plus) is clearly different from that corresponding to the rest of the bins (Fig. S10A).
Furthermore, we can easily identify those neurons participating in the encoding of each stimulus by averaging the importance maps across time and trials for each stimulus (see caption to Fig.
S2 for details).
As shown in Fig.
S2F, the neurons that (by design) participate in the encoding of a particular stimulus (indicated by red pluses and blue crosses for stim1 and stim2, respectively) are those presenting the highest importance values.
Given the clear bimodal distribution of the importance values shown in Fig.
S2F, we can easily set a threshold to select those neurons encoding each stimulus.
The recall and precision values (Ting, 2011) corresponding to this threshold is also shown in Fig.
S2F.
Finally, we have evaluated the importance maps in noisier scenarios, in which the spikes forming the packets present noise in their timing (Fig. S9) or in which the number of samples available to train
Spike-GAN is smaller (Fig. S10).
In both cases we have found that the importance maps analysis is able to provide highly valuable information about the neurons participating in the encoding of the stimulus information and their relative timing.
The features of neural activity individuated as important by the spike GAN importance maps may or may not be used by the animal to make a decision.
To test this causally, one can replace, during a perceptual discrimination task, the presentation of a sensory stimulus with the presentation of a 2Poptogenetic activation pattern.
Importantly, the experimenter can, in each stimulation trial, either keep or shufﬂe the features previously highlighted by the importance maps before imposing the pattern.
Comparing, between the altered and original patterns, the behavioral report of the animal about which stimulus was presented can then be used to ﬁnd out whether the information carried by each speciﬁc feature is used to produce a consistent behavioral percept.
For instance, a stimulation protocol in which only part of the neurons participating in a particular feature is stimulated would provide information about the capacity of the mouse brain to ’ﬁll in the gaps’.
Alternatively, the time at which each neuron is stimulated could be altered in order to study the role spike timing plays in the encoding of the stimulus.
[Figure]
l
[Figure]
nu m
[Figure]
r i c a l p r o b a
[Figure]
ili t y
[Figure] [Figure]