Original: E FFICIENT APPROXIMATION OF NEURAL POPULATION STRUCTURE AND CORRELATIONS WITH PROBABILISTIC CIRCUITS
Revised: E FFICIENT APPROXIMATION OF NEURAL POPULATION STRUCTURE AND CORRELATIONS WITH PROBABILISTIC CIRCUITS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a computationally efficient framework to model a wide range of population structures with high order correlations and a large number of neurons.
Revised: We present a computationally efficient framework to model a wide range of population structures with high order correlations and a large number of neurons.

Original: Our method is based on a special type of Bayesian network that has linear inference time and is founded upon the concept of contextual independence.
Revised: Our method is based on a special type of Bayesian network that has linear inference time and is founded upon the concept of contextual independence.

Original: Moreover, we use an efficient architecture learning method for network selection to model large neural populations even with a small amount of data.

Original: Our framework is both fast and accurate in approximating neural population structures.
Revised: Our framework is both fast and accurate in approximating neural population structures.

Original: Furthermore, our approach enables us to reliably quantify higher order neural correlations.
Revised: Furthermore, our approach enables us to reliably quantify higher order neural correlations.

Original: We test our method on simulated neural populations commonly used to generate higher order correlations, as well as on publicly available large-scale neural recordings from the Allen Brain Observatory.
Revised: We test our method on simulated neural populations commonly used to generate higher order correlations, as well as on publicly available large-scale neural recordings from the Allen Brain Observatory.

Original: Our approach significantly outperforms other models both in terms of statistical measures and alignment with experimental evidence
Revised: Our approach significantly outperforms other models both in terms of statistical measures and alignment with experimental evidence

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the rise and fast growth of simultaneous neural population recording, modeling population structures and measuring correlations has become a focus of computational neuroscience Abbott & Dayan (1999); Averbeck et al.
Revised: With the rise and fast growth of simultaneous neural population recording, modeling population structures and measuring correlations has become a focus of computational neuroscience Abbott & Dayan (1999); Averbeck et al.

Original: ; Azeredo da Silveira & Rieke (2021);
Revised: ; Azeredo da Silveira & Rieke (2021);

Original: Urai et al.
Revised: Urai et al.

Original: Theoretical and Experimental works have demonstrated the necessity of measuring population correlations to investigate information coding Moreno-Bote et al.
Revised: Theoretical and Experimental works have demonstrated the necessity of measuring population correlations to investigate information coding Moreno-Bote et al.

Original: (2014); Averbeck et al.
Revised: (2014); Averbeck et al.

Original: (2006), functional connectivity Dunn et al.
Revised: (2006), functional connectivity Dunn et al.

Original: , learning Ganmor et al.
Revised: , learning Ganmor et al.

Original: (2011), and arousal Vinck et al.
Revised: (2011), and arousal Vinck et al.

Original: ; Doiron et al.
Revised: ; Doiron et al.

Original: Despite significant progress in recent years, research on measurement and analysis of population correlations still faces significant challenges Kohn et al.
Revised: Despite significant progress in recent years, research on measurement and analysis of population correlations still faces significant challenges Kohn et al.

Original: Exact measurement of population correlations is an NP-hard problem in the general case since it requires computing every form of dependency among spiking neurons.
Revised: Exact measurement of population correlations is an NP-hard problem in the general case since it requires computing every form of dependency among spiking neurons.

Original: As a result, researchers have tried to come up with computationally efficient ways of approximation or indirect measurement of neural correlations.
Revised: As a result, researchers have tried to come up with computationally efficient ways of approximation or indirect measurement of neural correlations.

Original: Existing approaches are energy-based models rooted in statistical mechanics where the energy function incorporates couplings between subsets of variables (here neurons) Roudi et al.
Revised: Existing approaches are energy-based models rooted in statistical mechanics where the energy function incorporates couplings between subsets of variables (here neurons) Roudi et al.

Original: (2009c)
Revised: (2009c)

Original: ; Tkaˇcik et al.
Revised: ; Tkaˇcik et al.

Original: (2006); Sohl-Dickstein et al.
Revised: (2006); Sohl-Dickstein et al.

Original: ; Aurell & Ekeberg (2012).
Revised: ; Aurell & Ekeberg (2012).

Original: However, these methods often carry auxiliary (and even unrealistic) assumptions about the neural dynamics, do not scale up for large populations, or are limited to very specific situations Roudi et al.
Revised: However, these methods often carry auxiliary (and even unrealistic) assumptions about the neural dynamics, do not scale up for large populations, or are limited to very specific situations Roudi et al.

Original: (2009b);
Revised: (2009b);

Original: Averbeck et al.
Revised: Averbeck et al.

Original: Notably, generative models commonly used in other domains such as latent variable methods are often not applicable to neural populations as spiking neural data is discrete and sparse Zhao et al.
Revised: Notably, generative models commonly used in other domains such as latent variable methods are often not applicable to neural populations as spiking neural data is discrete and sparse Zhao et al.

Original: Furthermore, various parameters such as behavioural and emotional state of the animal affect firing patterns of neurons even in sensory cortex
Revised: Furthermore, various parameters such as behavioural and emotional state of the animal affect firing patterns of neurons even in sensory cortex

Original: Urai et al.
Revised: Urai et al.

Original: As a result, a recording long enough to train these models contains many external variable changes and confounding factors that make drawing scientific conclusions difficult.
Revised: As a result, a recording long enough to train these models contains many external variable changes and confounding factors that make drawing scientific conclusions difficult.

Original: In this paper, we take a probabilistic approach by modeling the joint probability distribution of neural activity with Bayesian networks.
Revised: In this paper, we take a probabilistic approach by modeling the joint probability distribution of neural activity with Bayesian networks.

Original: Inference is NP-hard in general/classic Bayesian networks Cooper (1990), making them impractical to model the population structure.
Revised: Inference is NP-hard in general Bayesian networks Cooper (1990), making them impractical to model the population structure.

Original: Therefore, we utilize a special family of Bayesian networks with linear inference time, first introduced as “Arithmetic circuits” Darwiche (2003);
Revised: Therefore, we utilize a special family of Bayesian networks with linear inference time, first introduced as “Arithmetic circuits” Darwiche (2003);

Original: Shen et al.
Revised: Shen et al.

Original: This family of networks has been designed to take advantage of “context-specific independence” of variables mainly for the purpose of computational efficacy, which also makes it suitable to extract local structures in the data Boutilier et al.
Revised: This family of networks has been designed to take advantage of “context-specific independence” of variables mainly for the purpose of computational efficacy, which also makes it suitable to extract local structures in the data Boutilier et al.

Original:  Shen et al.
Revised: ; Shen et al.

Original: We use a modification (and equivalent Rooshenas & Lowd (2014)) of arithmetic circuits, Sum-Product Networks (SPNs).
Revised: We use a modification (and equivalent Rooshenas & Lowd (2014)) of arithmetic circuits, Sum-Product Networks (SPNs).

Original: Sum product networks are more known and used by the community due to their success in image completion tasks Poon & Domingos (2011); Sanchez-Cauce et al.
Revised: SPNs are more known and used by the community due to their success in image completion tasks Poon & Domingos (2011); Sanchez-Cauce et al.

Original: In particular, we design a method that learns the architecture of the sum-product network from the spiking neural data in order to capture a wide range of population correlation/structure from local to global in polynomial time.
Revised: In particular, we adapt sum-product networks to fit spiking neural data in order to capture a wide range of population correlation/structure from local to global in polynomial time.

Original: Due to the efficiency ofour architecture learning and inference in SPNs, population structure estimation is polynomial in the size of the population.
Revised: Due to the efficiency of architecture learning and inference in SPNs, population structure estimation is polynomial in the size of the population.

Original: In addition, we suggest a measure of high order population correlations based on our framework.
Revised: In addition, we suggest a measure of high order population correlations based on our framework.

Original: Our results include fitting on neural population simulations constructed with higher order correlations, as well as large-scale neural recording in different brain regions on more than 20 mice.
Revised: Our results include fitting on neural population simulations constructed with higher order correlations, as well as large-scale neural recording in different brain regions on more than 20 mice.

Original: Our framework outperforms both of the best energy-based and latent variable models for neural population structure estimation.
Revised: Our framework outperforms both of the best energy-based and latent variable models for neural population structure estimation.

Original:  P ROBLEM D EFINITION AND R ELATED W ORK
Revised:  P ROBLEM D EFINITION AND R ELATED W ORK

Original: One of the critical problems in computational neuroscience is providing an accurate statistical description of spike trains in a population of neurons.
Revised: One of the critical problems in computational neuroscience is providing an accurate statistical description of spike trains in a population of neurons.

Original: As the full representation of the data, i.e. raw spike times, is high dimensional, spike trains are binned into small time windows.
Revised: As the full representation of the data, i.e. raw spike times, is high dimensional, spike trains are binned into small time windows.

Original: The time bin should be short enough so each neuron spikes at most once in each bin (with some amount of tolerance in potentially losing some spikes).
Revised: The time bin should be short enough so each neuron spikes at most once in each bin (with some amount of tolerance in potentially losing some spikes).

Original: In addition, this time bin should be large enough that the assumption of temporal independence of spikes holds.
Revised: In addition, this time bin should be large enough that the assumption of temporal independence of spikes holds.

Original: With this time binning strategy, each neuron’s activity is a binary variable ( S i for neuron i is equal to 1 if there is a spike in the corresponding bin, otherwise 0 ) and each time bin represents an i.i.d sample/instance.
Revised: With this time binning strategy, each neuron’s activity is a binary variable ( S i for neuron i is equal to 1 if there is a spike in the corresponding bin, otherwise 0 ) and each time bin represents an i.i.d sample/instance.

Original: Therefore, spike trains of N neurons for the duration of T would be represented as a binary matrix D K × N where K = T/ ∆ t in which ∆ t is the bin length (Figure 1, left plot).
Revised: Therefore, spike trains of N neurons for the duration of T would be represented as a binary matrix D K × N where K = T/ ∆ t in which ∆ t is the bin length (Figure 1, left plot).

Original: Consequently, the population activity has a probabilistic representationP ( S 1 , . . .
Revised: Consequently, the population activity has a probabilistic representation

Original:  , S N ) , and the problem turns into modelling this joint distribution, given the data.
Revised: P ( S 1 , . . . , S N ) , and the problem turns into modelling this joint distribution, given the data.

Original: More specifically, the problem is to find a model m ∗ from a family of models M , and optimize its free parameters Θ m so as to satisfy the following:
Revised: More specifically, the problem is to find a model m ∗ from a family of models M , and optimize its free parameters Θ m so as to satisfy the following:

Original: [Equation]
Revised: [Equation]

Original: In the existing approaches, M is set to maximum entropy (Ising) models, which are energy-based methods rooted in statistical mechanics Roudi et al.
Revised: In the existing approaches, M is set to maximum entropy (Ising) models, which are energy-based methods rooted in statistical mechanics Roudi et al.

Original: (2009c)
Revised: (2009c)

Original: ; Schneidman et al.
Revised: ; Schneidman et al.

Original: Since learning maximum entropy models is computationally very expensive, these models are restricted to estimate the statistical properties of the population up to a constant order.
Revised: Since learning maximum entropy models is computationally very expensive, these models are restricted to estimate the statistical properties of the population up to a constant order.

Original: However, going beyond second order is not computationally feasible.
Revised: However, going beyond second order is not computationally feasible.

Original: In fact, building the exact generative model is an intractable problem in the general case even for the second order (pairwise) correlations.
Revised: In fact, building the exact generative model is an intractable problem in the general case even for the second order (pairwise) correlations.

Original: Therefore, even a Pairwise Maximum Entropy (PME) model requires further estimation where more accurate approximation algorithms requires thousands of samples for each pair, making them impossible to be used for large populations of neurons Roudi et al.
Revised: Therefore, even a Pairwise Maximum Entropy (PME) model requires further estimation where more accurate approximation algorithms requires thousands of samples for each pair, making them impossible to be used for large populations of neurons Roudi et al.

Original: (2009c)
Revised: (2009c)

Original: ; Tkaˇcik et al.
Revised: ; Tkaˇcik et al.

Original: Sohl-Dickstein et al.
Revised: Sohl-Dickstein et al.

Original: ; Aurell & Ekeberg (2012).
Revised: ; Aurell & Ekeberg (2012).

Original: Moreover, there exist plausible scenarios, such as a dichotomized common input to loosely coupled neurons, in which pairwise correlations are negligible compared to higher order correlations Macke et al.
Revised: Moreover, there exist plausible scenarios, such as a dichotomized common input to loosely coupled neurons, in which pairwise correlations are negligible compared to higher order correlations Macke et al.

Revised: (2011b)

Original: ; Amari et al.
Revised: ; Amari et al.

Original: Overall,
Revised: Overall,

Original: PME is neither computationally efficient nor accurate in population structure estimation.
Revised: PME is neither computationally efficient nor accurate in population structure estimation.

Original: To improve the structure estimation (and also as further evidence for inaccuracy of PME models), some methods modelled higher order correlations through the energy based approaches, e.g. “kpairwise” correlationsTkaˇcik et al.
Revised: To improve the structure estimation, some methods modelled higher order correlations through the energy based approaches, e.g. “k-pairwise” correlationsTkaˇcik et al.

Original: , Restricted Boltzmann Machines (RBMs) on top of pair-wised correlations (semi RBMS (sRBMs))
Revised: Restricted Boltzmann Machines (RBMs) on top of pair-wised correlations (semi RBMS (sRBMs))

Original: Köster et al.
Revised: Köster et al.

Original: (2014), and sparse low-order correlations Ganmor et al.
Revised: (2014), and sparse low-order correlations Ganmor et al.

Original: Founded upon energy based models, these methods also needs computationally expensive sampling strategies to estimate the partition function.
Revised: Founded upon energy based models, these methods also needs computationally expensive sampling strategies to estimate the partition function.

Original: Moreover, to approximate the partition function, these methods often rely on pseudo-likelihoods which needs extensive amount of data points to be a accurate.
Revised: Moreover, to approximate the partition function, these methods often rely on pseudo-likelihoods which needs extensive amount of data points to be a accurate.

Original: Here we use a special type of Bayesian network with linear inference, and consequently fast learning time, to estimate distributions of joint neural activity.
Revised: Here we use a special type of Bayesian network with linear inference, and consequently fast learning time, to estimate distributions of joint neural activity.

Original: Our network approximates these distributions by ignoring correlations based on their effect size, rather than statistical order.
Revised: Our network approximates these distributions by ignoring correlations based on their effect size, rather than statistical order.

Original: Moreover, the normalization process to obtain real probability values is also linear in the size of the network.
Revised: Moreover, the normalization process to obtain real probability values is also linear in the size of the network.

Original: As a result, it is significantly more successful than energy-based models both in terms of efficiency and accuracy.
Revised: Thus, it is significantly more successful than energy-based models both in terms of efficiency and accuracy.

Original:  M ODEL
Revised:  M ODEL

Original: Our framework is based on Sum-Product Networks (SPNs).
Revised: Our framework is based on Sum-Product Networks (SPNs).

Original: An SPN is a rooted directed acyclic graph representing a joint probability distribution of given variables.
Revised: An SPN is a rooted directed acyclic graph representing a joint probability distribution of given variables.

Original: This distribution is the result of a hierarchical combination of alternating mixtures (sum nodes) and factorizations (product nodes), with given variables as the leaf nodes of the network Poon & Domingos (2011); Sanchez-Cauce et al.
Revised: This distribution is the result of a hierarchical combination of alternating mixtures (sum nodes) and factorizations (product nodes), with given variables as the leaf nodes of the network Poon & Domingos (2011); Sanchez-Cauce et al.

Original: Specifically, each leaf node (a node with no children) represents a univariate probability distribution.
Revised: Specifically, each leaf node (a node with no children) represents a univariate probability distribution.

Original: When the variable is categorical, the leaf node is a variable indicator ( I ( x ) ).
Revised: When the variable is categorical, the leaf node is a variable indicator ( I ( x ) ).

Original: The scope of each leaf node is a singleton where its element is the variable that the leaf represents.
Revised: The scope of each leaf node is a singleton where its element is the variable that the leaf represents.

Original: Other nodes are either sum or product .
Revised: Other nodes are either sum or product .

Original: A product node represents the product of its children (connected by a link).
Revised: A product node represents the product of its children (connected by a link).

Original: A sum node represents the (normalized) weighted sum of its children.
Revised: A sum node represents the (normalized) weighted sum of its children.

Original: The weight of each child is shown as the label of the link from the sum node to that child.
Revised: The weight of each child is shown as the label of the link from the sum node to that child.

Original: The scope of sum and product nodes is the union of scopes of their children.
Revised: The scope of sum and product nodes is the union of scopes of their children.

Original: Starting from a leaf as a one-node SPN, sum product networks can be built bottom-up by combining smaller SPNs through sum or product nodes.
Revised: Starting from a leaf as a one-node SPN, sum product networks can be built bottom-up by combining smaller SPNs through sum or product nodes.

Original: The root of the SPN represents a valid joint probability distribution if the SPN is complete and decomposable .
Revised: The root of the SPN represents a valid joint probability distribution if the SPN is complete and decomposable .

Original: An SPN is complete iff all children of each sum node have the same scope.
Revised: An SPN is complete iff all children of each sum node have the same scope.

Original: A sum-product network is decomposable iff all children of each product node have disjoint scopes.
Revised: A sum-product network is decomposable iff all children of each product node have disjoint scopes.

Original: Parameter Learning :
Revised: Parameter Learning :

Original: To model the joint probability distribution of the given variables, the weights on the links of sum nodes should be learned with a learning algorithm such as gradient descent or Expectation-Maximization (EM), given the data
Revised: To model the joint probability distribution of the given variables, the weights on the links of sum nodes should be learned with a learning algorithm such as gradient descent or Expectation-Maximization (EM), given the data

Original: Sanchez-Cauce et al.
Revised: Sanchez-Cauce et al.

Original: Similar to other
Revised: Similar to other

Original: Bayesian networks, the structure of the graph needs to be determined before parameter learning.
Revised: Bayesian networks, the structure of the graph needs to be determined before parameter learning.

Original: One approach is to use a random dense graph and rely on the parameter learning algorithm to select important components.
Revised: One approach is to use a random dense graph and rely on the parameter learning algorithm to select important components.

Original: This strategy requires data sets orders of magnitudes larger than that gathered in neural experiments.
Revised: This strategy requires data sets orders of magnitudes larger than that gathered in neural experiments.

Original: Structure Learning : Network structure can also be learned from the data directly based on the general computational properties of sum and product nodes Gens & Pedro (2013).
Revised: Structure Learning : Network structure can also be learned from the data directly based on the general computational properties of sum and product nodes Gens & Pedro (2013).

Original: Sum nodes represent the sum of probability distributions over the same set of variables (see completeness above).
Revised: Sum nodes represent the sum of probability distributions over the same set of variables (see completeness above).

Original: This means that they act as a union of sets.

Original: Therefore, they explain the data best if their child distributions represent disjoint sets of data (instances), or in other words different “contexts” in the data.
Revised: Therefore, they explain the data best if their child distributions represent disjoint states, or in other words different “contexts” in the data.

Original: A Product node computes the joint distribution of its children with a product, explicitly carrying an assumption of independence among the factors contributing to the product.
Revised: A Product node computes the joint distribution of its children with a product, explicitly carrying an assumption of independence among the factors contributing to the product.

Original: This means that recursive splittings of the data based on instances (e.g. by  a clustering method) producing a sum node, and based on variables (e.g. via performing independence tests) resulting in a product node would give us a suitable SPN structure for the given data.
Revised: Children of a product node can be determined by using a dependence/correlation measure such as mutual information as a link/edge between each pair of features, subjected to a threshold. In other words, to make a tree/sub-tree with a product node as the root, a graph (totally different from the main SPN tree) is built where two feature nodes are connected if and only if they are considered dependent according to a criterion.

Original: While there exist few algorithms for structure learning Gens &
Revised: Each connected component of this graph is a sub-tree in the sum-product architecture where its root is linked to the root product node. Picturing the data in matrix form where each row is an instance (or sample) and each column represents a feature (or variable), one can think of the action of a sum node as splitting the rows (samples) and that of a product node as splitting the columns (features). This means that recursive splittings of data rows (e.g. by a common clustering method) producing a sum node, and based on variables (e.g. via performing independence tests) resulting in a product node would give us a suitable

Original:  Pedro (2013); Vergari et al.(2015), we developed our own method due to our specific aim which is learning correlations in neural population.
Revised: SPN structure for the given data Gens & Pedro (2013); Vergari et al.

Original: 3. N EURAL POPULATION ANALYSIS WITH nSPN S
Revised: 3. N EURAL POPULATION ANALYSIS WITH nSPN S

Original: We construct a family of SPNs that we call Neural SPNs ( nSPN ) to model the joint probability distribution of spiking in populations of neurons.
Revised: We construct a family of SPNs that we call Neural SPNs ( nSPN ) to model the joint probability distribution of spiking in populations of neurons.

Original: These models are represented by directed graphs where leaves of an nSPN representing spiking activity of N neurons correspond to I s i and I ¯
Revised: These models are represented by directed graphs where leaves of an nSPN representing spiking activity of N neurons correspond to I s i and I ¯

Original: s i ( 1 ≤ i ≤ N ) indicating whether neuron i fired or not in a given time bin (see section 2).
Revised: s i ( 1 ≤ i ≤ N ) indicating whether neuron i fired or not in a given time bin (see section 2).

Original: We suggest a systematic way to build the overall architecture of  an nSPN suitable to capture population correlations even with a limited number of data points.
Revised: Shallow architecture of nSPN s: The simplest structure of an nSPN consists of a product node as a root, linked to N sum nodes. Each sum node represents the spike probability of a neuron i , linked to I s i and I

Original: First we describe a shallow nSPN with three layers and then a mechanism to construct deep architectures that impose regularization for better
Revised: ¯ s i leaf nodes. This model, shown as nSPN 1 is basically a Naive Bayes model, assuming complete independence among neurons. The right plot in figure 1 shows an example

Original: × + + + ....
Revised: × + + + ....

Original: I S 

Revised: I S

Original: I ¯ S 1 I S 
[-I ¯ S I S 
-]
Revised: I ¯ S 1 I S 
[-I ¯ S I S 
-]

Original: I ¯ S..  ..  .. fits. These deep architectures are obtained by clustering based on the mutual information between different neurons. Finally, we describe our measure of high order correlations and describe a mixture of nSPN models that aims to capture correlations with deep structures. Shallow architecture of nSPN s: The simplest structure of an nSPN (in terms of graph complexity) consists of a product node as a root, linked to N sum nodes.
Revised: I ¯ S

Original: Each sum node represents the spike probability of a neuron i , linked to I s i and I ¯ s i leaf nodes. This model, shown as nSPN 1 is basically a Naive Bayes model, assuming complete independence among neurons.{+S 
+}
Revised: I {+S 
+}

Original: The right plot in figureshows an example nSPN 1 trained on the joint activity of three neurons in the left plot. As mentioned before, each node of an SPN represents the probability of the leaves of the (sub-)tree of which it is the root. As a result, (normalized) link weights of each sum node are aligned with the mean spike probability of the related neuron (note the assumption of independence). Furthermore, given an assignment of probabilities to the leaf nodes and a configurationof spiking activity, the root represents the probability of that configuration’s occurrence according to the model (shown by the red number near the root node).
Revised: I ¯ S ..  ..  .. configuration

Original: Within the SPN framework, a simple extension of the Naive Bayes model involves the introduction of multiple contexts , with population spiking being described by a Naive Bayes assumption conditioned on the context .
Revised: Within the SPN framework a simple extension of the Naive Bayes model involves the introduction of multiple contexts , with population spiking being described by a Naive Bayes assumption conditioned on the context .

Original: If there are two such contexts with associated probability, the model consists of a sum node as the root linked to two product nodes, each of which a root of an nSPN 1 (sub)-tree.
Revised: If there are two such contexts with associated probability, the model consists of a sum node as the root linked to two product nodes, each of which a root of an nSPN 1 (sub)-tree.

Original: Similarly, a model can include an arbitrary number of contexts by increasing the number of product nodes under the root, e.g. nSPN b for b product nodes/contexts.
Revised: Similarly, a model can include an arbitrary number of contexts by increasing the number of product nodes under the root, e.g. nSPN b for b product nodes/contexts.

Original: In principle, one can construct a shallow graph sufficient to model anyneural data set by adding enough product nodes.
Revised: In principle, one can construct a shallow graph sufficient to model any data set by adding enough product nodes.

Original: One problem with this approach is that there exist simple networks with deeper structure for which a three-layer model like we have described will require an exponential number of nodes Delalleau & Bengio (2011), which in addition to tractability also raises the issue of data limitations.
Revised: One problem with this approach is that there exist simple networks with deeper structure for which a 3-layer model like we have described will require an exponential number of nodes Delalleau & Bengio (2011), which in addition to tractability also raises the issue of data limitations.

Original: Moreover, all existing parameter learning algorithms for SPNs find local optima and shallow networks get stuck in local optima more frequently.
Revised: Moreover, all existing parameter learning algorithms for SPNs find local optima and shallow networks get stuck in local optima more frequently.

Original: Most importantly, when the data is limited deeper architectures of our framework avoid over-fitting as their structure implements regularization.
Revised: Most importantly, when the data is limited deeper architectures of our framework avoid over-fitting as their structure implements regularization.

Original: Deep architecture of nSPN s: As computing all correlations is intractable, we seek an approach for constructing graphs that trades some fidelity for efficiency.
Revised: Deep architecture of nSPN s: As computing all correlations is intractable, we seek an approach for constructing graphs that trades some fidelity for efficiency.

Original: Our approach is to identify correlated ensembles of neurons and build deeper graphs that reflect these ensembles.

Original: Since product nodes in

Original: SPNs describe a factorization into independent distributions with separate scopes, a natural approach, given a scope of neurons { S 1 , . . . , S N } , is to assign neurons to a number of groups based on a clustering algorithm, and form a model that has a product node as the root, with children that form the models for each group in the clustering. Consider that such a clustering says to divide the neurons { S 1 , . . S N } into { S 1 , . .
Revised: SPN structure learning algorithms naturally achieve that by separating low correlation variables (according to an independence test and threshold on its p -value) at a product node.

Original: S j } and { S j +1 , . . .

Original: , S N } . One can then choose, for example, to model each group via the graphs nSPN 2 ( S 1 , . . . , S j ) and nSPN 2 ( S j +1 , . . . , S N ) .
Revised: Importantly, in deep layers of an SPN, variables are considered independent in a given context.

Original: This model would

Original: × + + × + + ...

Original: I S I ¯ S I S j I ¯ S j × + + ...
Revised: We utilized learnSPN Vergari et al.

Original: I S

Original: I ¯ S

Original: I S j

Original: I ¯ S j × + + ...

Original: I S

Original: [-j +
-]
Revised: [-j +
-]

Original: I ¯ S j +

Original: I S N

Original: I ¯ S N × + + ...

Original: I S

Original: [-j +
 I ¯ S j + I S N I ¯ S N give the graph-] shown in Figure 2.
Revised: [-j +
 I ¯ S j + I S N I ¯ S N give the graph-](2015), shown by

Original: Roughly speaking, this model describes j neurons with two common contexts and N − j neurons with two different contexts that vary independently of those for the j neurons.

Original: We can iterate this process to produce a graph with a finer scale.

Original: Instead of using nSPN 2 to describe the distribution of { S 1 , . . . , S j } we can iterate the clustering approach and separate these into { S 1 , . . . , S k } and { S k +1 , . . . , S j } .
Revised: SPN l , due to its more conservative approach for splitting.

Original: These in turn can either be described by an nSPN b model or we can continue separating into clusters.

Original: For example, nSPN bd is an SPN in which d levels of clustering by product nodes is performed (the tree has 2d + 3 layers), and each sum node connected to intermediate nodes has b branches.

Original: Note that, one can easily use different number of branches for each node, or another criterion for splitting the variables without loss of generality (e.g minimum scope size for the clustering).

Original: Correlation-based clustering of neurons

Original: : A product node assumes its children are independent.

Original: Therefore, to lose the least amount of population correlation, we would ideally minimize the mutual information between the two groups Gens & Pedro (2013).

Original: Formally, splitting a variable set S into nonempty subsets of V and S \ V is optimal

Original: iff V = arg

Original: min

Original: X ⊂ S I

Original: ( X, S \ X )

Original: where I ( X, Y ) = H ( X ) + H ( Y ) − H ( X, Y ) and H ( X ) = − (cid:80) ni =1 P ( x i ) log P ( x i ) .
Revised: This method has two hyper-parameters, an independence threshold and a minimum number of data instances to continue splitting.

Original: Since mutual information is a sub-modular function, the optimal split can be achieved in polynomial time by Queyranne’s algorithm if querying H ( X, Y ) is in polynomial time, e.g. through an oracle Queyranne (1998); Narasimhan et al. ; Gens & Pedro (2013); Hidaka & Oizumi (2018). This is not the case for us.
Revised: These hyper parameters can be tuned either by a cross validation procedure or based on the data size.

Original: In fact, such query has the exact nature of our main goal, i.e. estimating the joint probability of  the set. Therefore, we use the marginal mutual information between pairs, I ( s i , s j )(1 ≤ i, j ≤ N ) instead. With such a pairwise marginal entropy estimate, approaches that are based on maximum/minimum linkage criterion such as minimum/maximum spanning trees are the most suitable. Particularly, we use agglomerative hierarchical clustering with the distance metric of log(2)
Revised: Notably, due to our small data size, we applied a Laplace-like regularization to the method to prevent pair-weights of [0, 1] (absolute zero makes the log-likelihood -inf, so we regularized it to [10 − 5 , 1 − 10 − 5 ] .)

Original: − I ( s i , s j ) between variable i and j and linkage criterion of average distance.

Original: Measure of Neural Correlation and mixture-of- nSPNs : The simplest architecture, nSPN 1 , assumes independence between all neurons.
Revised: Measure of Neural Correlation: The simplest architecture, nSPN 1 , assumes independence between all neurons.

Original: On the other hand, the more complicated structures in our framework only assume independence within each context where the number of contexts are always more than 1.
Revised: On the other hand, the more complicated structures in our framework only assume independence within each context where the number of contexts are always more than 1.

Original: This means that in stationary periods where the firing patterns of neurons do not change significantly, nSPN 1 fits to the data as well as more complexnSPN structures if and only of all neurons are independent of each other.
Revised: This means that in stationary periods where the firing patterns of neurons do not change significantly, nSPN 1 fits to the data as well as more complex structures if and only if all neurons are independent of each other.

Original: Any improvement in goodness of fit for more complicated structures(relative to nSPN 1 ) is due to the existence of neural correlations.
Revised: Any improvement in goodness of fit for more complicated structures is due to the existence of neural correlations.

Original: As a result, we can use the difference between the average log-likelihood (equation 1) of the best fitting nSPN and nSPN 1 as a measure for neural correlations.
Revised: As a result, we can use the difference between the average log-likelihood (equation 1) of the best fitting nSPN and nSPN 1 as a measure for neural correlations.

Original: For example, if our best model is nSPNb ∗ , our measure for population neural correlation is:
Revised: For example, if our best model is nSPN ∗ , our measure for population neural correlation is ∆ ll

Original: [Equation]
Revised: [Equation]

Original: Any shallow structure SPN, i.e. nSPN b , is a super-set of nSPN 1 .

Original: This is not true for deep architectures though.

Original: As a result, especially for the purpose of measuring population correlations, a mixture of deep and shallow architectures should be used, e.g. a model constructed with a sum node as the root connected to a shallow structure such as nSPN 1 and a deep one such as nSPN bd .

Original: We call this structure mixture-of- nSPN s.

Original:  R ESULTS
Revised:  R ESULTS

Original: We start with a simulation of population activity in which we can control the presence of fluctuations.
Revised: We start with a simulation of population activity in which we can control the presence of fluctuations.

Original: We use a network of homogeneous exponential integrate-and-fire (EIF) neurons with common fluctuating input where each neuron’s membrane voltage, V i (1 ≤ i ≤ N ) evolves as follows:
Revised: We use a network of homogeneous exponential integrate-and-fire (EIF) neurons with common fluctuating input where each neuron’s membrane voltage, V i (1 ≤ i ≤ N ) evolves as follows:

Original: [Equation]
Revised: [Equation]

Original: Notably, it has been shown that pairwise maximum entropy (PME) models are incapable of estimating this network very well especially when the firing rate of neurons is low, neural correlations are high, or the population size is large Amari et al.
Revised: Notably, it has been shown that pairwise maximum entropy (PME) models are incapable of estimating this network very well especially when the firing rate of neurons is low, neural correlations are high, or the population size is large Amari et al.

Original: ; Macke et al. 
Revised: ; Macke et al. (2011b); Leen & Shea-Brown (2015).

Original: Moreover, experimental studies support the existence of networks producing extensive high order correlations in the brain
Revised: Moreover, experimental studies show the existence of higher order correlations in the brain

Original: Binder & Powers (2001); Khuc-Trong & Rieke (2008).
Revised: Binder & Powers (2001); Khuc-Trong & Rieke (2008).

Original: Therefore, the network above has been used as a tool to assess the performance of methods for capturing beyond pairwise correlations Macke et al.
Revised: Therefore, the network above has been used as a tool to assess the performance of methods for capturing beyond pairwise correlations Macke et al.

Original: ; Leen & Shea-Brown (2015).
Revised: (2011b); Leen & Shea-Brown (2015).

Original: Similarly, we generate multiple data sets using varying parameters to get different values for the mean activity µ and (pairwise) correlation ρ and compare the goodness of fit between nSPN N and PME models for population size of N (all parameter values in Supplementary Material).
Revised: Similarly, we generate multiple data sets using varying parameters to get different values for the mean activity µ and (pairwise) correlation ρ and compare the goodness of fit between nSPN N/ 2 and PME models for population size of N (all parameter values in Supplementary Material).

Original: Note that nSPN N is chosen to have the same number number of parameters as the PME, N  2 ( nSPN N has N 2  − 1 free parameters to be more precise).
Revised: Note that nSPN N/ 2 is chosen to have the same number number of parameters as the PME, N ( N + 1) / 2 ( nSPN N/ 2 has N 2 / 2 − 1 free parameters to be more precise).

Original: Pairwise Maximum Entropy models have minimal assumptions about the spiking pattern of the population beyond the firing rate of each neuron and the relationship between each two neurons.
Revised: Pairwise Maximum Entropy models have minimal assumptions about the spiking pattern of the population beyond the firing rate of each neuron and the relationship between each two neurons.

Original: In full generality, a PME model is complex with many parameters, and is computationally expensive in general.
Revised: In full generality, a PME model is complex with many parameters, and is computationally expensive in general.

Original: In fact, as mentioned before, an exact fit of PME is NP-hard.
Revised: In fact, as mentioned before, an exact fit of PME is NP-hard.

Original: Therefore,  instead of thewhole joint probability distribution we used the (total) populationactivity as the measure of goodness of fit. According to the PME, the probability of observing overall k spikes  in a population of size N is:
Revised: Therefore, we instead fit to the probability that the a population of N neurons will generate k spikes overall in a single time bin. According to the PME model, this is given by

Original: [Equation]
Revised: [Equation]

Original: where α and β are free parameters, obtained from fitting the model to population activity.
Revised: where α and β are free parameters, obtained from fitting the model to population activity.

Original: Importantly, instead of fitting the PME to the whole joint distribution we obtained α and β by directly optimizing the above equation.
Revised: Importantly, instead of fitting the PME to the whole joint distribution we obtained α and β by directly optimizing the above equation.

Original: This measure reflects the maximum possible power of any PME model without any need to actually obtain all variables (it essentially provides a bound on the performance of the full model).
Revised: This measure reflects the maximum possible power of any PME model without any need to actually obtain all variables (it essentially provides a bound on the performance of the full model).

Original: Note that, the distribution of average population activity fully reflects the goodness of fit in fully homogeneous populations Roudi et al.
Revised: Note that, the distribution of average population activity fully reflects the goodness of fit in fully homogeneous populations Roudi et al.

Original: (2009a).
Revised: (2009a).

Original: However, there is no guarantee that existing approximation methods for PMEs achieve such fit.
Revised: However, there is no guarantee that existing approximation methods for PMEs achieve such fit.

Original: As opposed to the PME, we fitted an nSPNN to the whole joint distribution (200 iterations of
Revised: As opposed to the PME, we fitted an nSPN to the whole joint distribution (200 iterations of

Original: Expectation Maximization (EM) implemented in SPFlow library Molina et al.
Revised: Expectation Maximization (EM) implemented in SPFlow library Molina et al.

Original: , giving the PME a huge advantage .
Revised: , giving the PME a huge advantage.

Original: Specifically, the difference between the model’s prediction and the
Revised: Specifically, the difference between the model’s prediction and the

Original: EIF (total) population spike distribution is quantified with the Jenson-Shannon (JS)-divergence of2 D (cid:0) P model ( k )
Revised: EIF (total) population spike distribution is quantified with the Jenson-Shannon (JS)-divergence of 12 D (cid:0) P model ( k )

Original: ∥ M ( k ) (cid:1) + 1 2 D (cid:0) P EIF ( k ) ∥
Revised: ∥ M ( k ) (cid:1) + 12 D (cid:0) P EIF ( k ) ∥

Original: M ( k ) (cid:1) where M is the averaged distribution, i.e. M ( k ) =
Revised: M ( k ) (cid:1) where M is the averaged distribution, i.e. M ( k ) =

Original: 12 P model ( k ) + 12 P EIF ( k ) and D ( . || . ) is the Kullback–Leibler (KL)-divergence.
Revised: 12 P model ( k ) + 12 P EIF ( k ) and D ( . || . ) is the Kullback–Leibler (KL)-divergence.

Original: Since JS-div scales with log( N ) , we report the normalized value, i.e. JS-div /log ( N ) .
Revised: Since JS-div scales with log( N ) , we report the normalized value, i.e. JS-div /log ( N ) .

Original: We set λ = .
Revised: We set λ = .

Original: 30 and . 59 to make the pairwise-correlation ρ equal to .
Revised: 30 and . 59 to make the pairwise-correlation ρ equal to .

Original: 1 and . 25 respectively, keeping other parameters the same as above.
Revised: 1 and . 25 respectively, keeping other parameters the same as above.

Original: In both cases, the mean activity rate µ is .
Revised: In both cases, the mean activity rate µ is .

Original: The number of neurons in the network is
Revised: The number of neurons in the network is

Original: N ∈ [8 , 16 , 32 , 64 , 128] , and each data point is obtained from  100000 samples.
Revised: N ∈ [8 , 16 , 32 , 64 , 128] , and each data point is obtained from training on 100000 and testing on another 100000 samples.

Original: As shown in the left panel of figure 3, nSPN models the EIF network significantly better than PME (the JS-divergence is orders of magnitude lower) especially when the number of neurons  are high.
Revised: As shown in the left panel of figure 2, nSPN models the EIF network significantly better than PME (the JS-divergence is orders of magnitude lower) especially when the number of neurons and correlation ( λ ) are high (dotted lines λ = . 59 , solid lines λ = . 3 ).

Original: We also compared our method to other approaches for capturing joint probability distributions of spiking activity.
Revised: We also compared our method to other approaches for capturing joint probability distributions of spiking activity.

Original: We trained a  Restricted Boltzmann Machine (RBM) with N visible and N hidden units ( N 2  free parameters).
Revised: We trained a Bernoulli Restricted Boltzmann Machine (RBM) with N visible and N/ 2 hidden units ( N 2 / 2 free parameters).

Original: The JS-div of the RBM was an order of magnitude higher than the SPN (figure 3, left plot).
Revised: The JS-div of the RBM was an order of magnitude higher than the SPN (figure 2, left plot).

Revised: Moreover, the SPN scales better with the increase of N (Note that the y-axis is in log-scale).

Original: Importantly, the RBM is the best known latent-variable model in capturing population correlations Köster et al.
Revised: Importantly, the RBM is the best known latent-variable model in capturing population correlations Köster et al.

Original: Moreover, adding energy-based component to a PME (called sRBM) does not provide a better fit to the neural population despite adding a huge amount of computation to the method Köster et al.
Revised: Moreover, adding energy-based component to a PME (called sRBM) does not provide a better fit to the neural population despite adding a huge amount of computation to the method Köster et al.

Revised: One of the goals of these benchmarks is to assess model performance as a function of population size ( N ).

Revised: Hence in some simulations we include a single, global input to all neurons in the population (governed by parameter λ in the equation below).

Revised: In addition, we introduce correlated subpopulations by adding a common input to specific subpopulation both pairwise (via λ 2 ) and in quadruples (via λ 4 )

Revised: [Equation]

Original: To further demonstrate that our model’s main target is to capture the largest correlations with its available resources (free parameters), we compared the shallow SPN with 2product nodes ( nSPN 2  )  with an RBM with two hidden nodes.
Revised: For simplicity we assumed these common inputs are among adjacent nodes: ∀ 1 ≤ k ≤ N/ 2 : ξ 2 k − 1 , 2 ( t ) = ξ 2 k, 2 ( t ) , and ∀ 1 ≤ k ≤ N/ 4 : ξ 4 k − 3 , 2 ( t ) = · · · = ξ 4 k, 2 ( t ) . We repeated the previous set of fits ( λ = . 3 ), this time with λ 2 = λ

Original: The RBM performs better with smile size populations.
Revised: 4 = . As demonstrated in figure 2 middle left plot, SPN outperformed the other models by orders of magnitude again. We also performed simulations with heterogeneous inputs, resulting in heterogeneous population activity.

Original: However,{+Neurons are divided into four groups with different voltage reset values and different values of λ (values can be found in the supplementary material). This network was simluated with both λ 2 = λ 4 = 
 (Figure 2 middle-right: dotted lines) and λ 2 = λ+}
Revised: {+Neurons are divided into four groups with different voltage reset values and different values of λ (values can be found in the supplementary material). This network was simluated with both λ 2 = λ 4 = 
 (Figure 2 middle-right: dotted lines) and λ 2 = λ+}

Original: SPN’s relative performance to RBM increases with population size as it eventually becomes superior for population size large than 128 (Figure 3, middle plot).
Revised: 4 = . (Figure 2 middle-right: solid lines). Our SPN significantly outperformed the other models. Note that due to the heterogeneous network activity, the PME direct fitting results are not a valid lower-bound for JS-div of a pairwise model anymore.

Original: Finally, we simulated N = 30 neurons with the original EIF dynamics and parameters, with different values of λ = [0 , .
Revised: We also tested recurrent networks with heterogeneous connectivity. We analyzed a well-known connected “balanced" network of excitatory and inhibitory spiking neurons Brunel (2000). The dynamics of the membrane voltage of each neuron in this network is: [Equation]

Revised: where I i ( t ) are the synaptic currents caused by spikes arriving at synapses of neuron i (more details in the supplementary material).

Original: Fitted on nSPN N and nSPN 1 (Naive Bayes) models, their average log-likelihood difference ( ∆ ll SPN = ll nSPN N − ll nSPN 1 ) increases with γ , confirming that our measurement is reflective of correlations (figure 3, right panel).
Revised: We specifically fit the models to two stable setups (figure 8, c and d) of the original paper with “stationary” and “slow oscillation” global activity Brunel (2000). As shown in the right plot of figure 2, JS-div of the SPN was orders of magnitude smaller than other methods. Importantly, here the size of the networks were always 1000, and each fitting was on the first N excitatory neurons. Moreover, similar to our previous network, JS-div from fitting PME directly to the population activity is no longer a valid lower-bound for pairwise models

Original:  E XPERIMENTAL NEURAL RECORDINGS
Revised: . E XPERIMENTAL NEURAL RECORDINGS

Original: To assess our method on experimental neural recordings we modelled the neural data from the Neuropixels Visual Coding data set of the Allen Brain Observatory (https://observatory.brain-map.org).
Revised: To assess our method on experimental neural recordings we modelled the neural data from the Neuropixels Visual Coding data set of the Allen Brain Observatory (https://observatory.brain-map.org).

Original: In this data set, different stimuli (Gabors, flashes, drifting gratings, etc) were shown to 26 mice while the neural activity of their visual cortical and sub-cortical regions were recorded simultaneously with multiple Neuropixels probes.
Revised: In this data set different stimuli (Gabors, flashes, drifting gratings, etc) were shown to 26 mice while the neural activity of their visual cortical and sub-cortical regions were recorded simultaneously with multiple Neuropixels probes.

Original: Mice were head-fixed and on a wheel, free to run.
Revised: Mice were head-fixed and on a wheel, free to run.

Original: Importantly, stimuli in this experiment were passively viewed, meaning that it did not involve any choice or reward.
Revised: Importantly, stimuli in this experiment were passively viewed, meaning that it did not involve any choice or reward.

Original: We modelled the neural population structure during the viewing of drifting gratings in 4 directions ( [0 ◦ , 45 ◦ , 90 ◦ , 135 ◦ ] ) with the contrast level of . 9 .
Revised: We modelled the neural population structure during the viewing of drifting gratings in 4 directions ( [0 ◦ , 45 ◦ , 90 ◦ , 135 ◦ ] ) with the contrast level of . 9 .

Original: Each of these 4 stimuli was shown to 26 micetimes.
Revised: Each of these 4 stimuli was shown to 26 micetimes.

Original: We analyzed the neural activity (spikes) of six regions in the visual cortex: VISp, VISl, VISal,
Revised: We analyzed the neural activity (spikes) of six regions in the visual cortex: VISp, VISl, VISal,

Original: VISpm, VISrl, VISam.
Revised: VISpm, VISrl, VISam.

Original: Following other studies analyzing neural correlations, the time bin was set to ∆ t = 20 msc, balancing for minimizing spike loss due to binning, and accounting for activity delays between neurons.
Revised: Following other studies analyzing neural correlations Köster et al. Tka ˇ cik et al. (2014), the time bin was set to ∆ t = 20 ms, balancing for minimizing spike loss due to binning and accounting for activity delays between neurons.

Original: We analyzed each of these 6 regions separately.
Revised: We analyzed each of these 6 regions separately.

Original: Each stimulus duration was 2 seconds.
Revised: Each stimulus duration was 2 seconds.

Original: We analyze the response from 400 ms after stimulus onset to 800 ms after stimulus onset.
Revised: We analyze the response from 400 ms after stimulus onset to 800 ms after stimulus onset.

Original: The delay is to make sure the corresponding signal has reached the neurons in all areas.
Revised: The delay is to make sure the corresponding signal has reached the neurons in all areas.

Original: Moreover, individual neurons’ change upon exposure to the same sensory stimuli (known as adaptation), hence the short period.

Original: Since 400 ms results in only 20 data points, we combined the data of 5consequent presentations with the same direction.
Revised: For each grating direction, we concatenated data from 5 presentations of 400 ms to get 100 data points.

Original: The order of stimulus condition (direction) of the stimulus type/block (drifting gratings) is random in the experiment.
Revised: The order of presentation of grating directions is random.

Original: We chose only 5 consequent presentations to keep the  state  of the animal (such as running speed or arousal level) as unchanged as possible.
Revised: In order to minimize the effects of state changes in the animal, we concatenate trials from subsequent presentations of the same grating direction.

Original: The described data processing results in 15 blocks of population activity per each tuple (mouse, direction, area), each of which we modelled with amixture-of- nSPN s (mixture of nSPN 1 and nSPN 21 ; details supplementary material).
Revised: This data processing results in 15 blocks of population activity per each tuple (mouse, direction, area), each of which we modelled with a nSPN l .

Original: We also fit our simplest structure nSPN 1 (Naive Bayes) in order to have a correlation-free baseline and compute the difference, ∆ ll SPN .
Revised: We also fit our simplest structure nSPN 1 (Naive Bayes) in order to have a correlation-free baseline and compute the difference, ∆ ll SPN .

Original: We fit both models on a modified version of data in which we shuffle the spikes of each neuron in the 400 ms of single stimulus showing for the primary visual cortical population (as the most sensitive area to events).
Revised: We fit both models on a modified version of data in which we shuffle the spikes of each neuron in the 400 ms of single stimulus showing for the primary visual cortical population (as the most sensitive area to events).

Original: We observed a large difference between ∆ ll SPN in the original and shuffled data.
Revised: We observed a large difference between ∆ ll SPN in the original and shuffled data.

Original: This difference confirmed our assumption about ∆ SPN in the original data is  thesole product of neural correlations (Figure 4 left).
Revised: This difference confirmed our assumption about ∆ SPN in the original data is mostly the product of neural correlations (Figure 3 left).

Original: For an extra measure of caution we removed (mouse, direction) pairs with ∆ ll SPN of more than .

Original: 1 in the primary visual area for the shuffled data.

Original: Since the population is not homogeneous, this time we used Minimum Flow Probability (MFP) SohlDickstein et al.
Revised: Since the population is not homogeneous, this time we used Minimum Flow Probability (MFP) Sohl-Dickstein et al.

Original: (2011) that approximates pairwise maximum entropy variables through a direct fit to the data.
Revised: (2011) that approximates pairwise maximum entropy variables through a direct fit to the data (implementation of Lee & Daniels (2019)).

Original: To the best of our knowledge this is the most accurate methods of pairwise approximation.
Revised: To the best of our knowledge this is the most accurate methods of pairwise approximation.

Original: First, tofurther demonstrate the computational efficiency of our method, we compared the training time of SPN and MFP approaches for 16 example data blocks of different sessions (area V ISp ) on the same machine. As demonstrated in the middle plot of figure 4, training time for the MFP is two orders of magnitude longer. Importantly, for MFP we used an efficient package that takes advantage of a C++ implementation in Python Lee & Daniels (2019), while the SPN was fully implemented in
Revised: We also fitted an RBM to the data. The number of hidden units in the RBM was determined through 10-fold cross-validation within the 100 samples (see supplementary material for details of fittings process for all models).

Original: Python Molina et al.

Original: The (total) population activity is not an ideal measure for performing model comparison.
Revised: The (total) population activity is not an ideal measure for performing model comparison.

Original: However, given the number of samples, i.e. 100, more desirable measures are prohibitive.
Revised: However, given the number of samples, i.e. 100, more desirable measures are prohibitive.

Original: We also fitted an

Original: RBM to the data.

Original: The number of hidden units in the RBM, and the depth added to the mixtureof nSPN s ( nSPN 1 and nsPN 21 ) were both determined through 10-fold cross-validation within the 100 samples. Compared to other models,the mixture-of- nSPNs produced significantly lower normalized JS-div for all of the 6 regions (Figure 4, right plot). 
Revised: Compared to other models, nSPN produced significantly lower normalized JS-div for all of the 6 regions (Figure 3, middle plot). In fact, J-div of nSPN was lower than other methods for nearly every single (animal, area, direction). The right plot in figure 3 shows this comparison v.s. RBM.

Original: We also performed a Leave One Out Cross Validation (LOOCV) test, in which the samples of each of the 15 blocks is generated based on  the otherfor each animal and direction (the validation set is outside of the fitted 100 samples and therefore unseen by the models).
Revised: We also tested our model on four short 340 ms) movie clips, cut from the “natural movie one” stimuli of the same data set, which is a one-shot 30 second video repeated 60 times.

Original: For the SPN, the JS-div was also significantly lower than the MPF, in allregions.
Revised: The frame rate is

Original: However, the cross validated JS-div for the RBM was extremely high, showing over-fitting to the data.
Revised: Hz, making each of our clip consists of 10 frames, starting from frame number 240, 390, 490 and 890. Similar to the previous stimulus, we grouped 5 consequent clips together, which resulted in 12 blocks of population activity. The time bin was set to 17 ms, meaning that each block included 100 data points.

Original: Importantly, the non-cross validated result shows that even when over-fitted the RBM performed worse than the SPN.
Revised: We repeated the analysis described above for this stimulus type and got consistent results (Figure 4). Notably, ∆ ll SPN was higher in general in this stimulus, probably because of longer time difference (at least 30 seconds) among the 5 trials of each group. Because of this, very computational cost, and poor results on a simpler stimulus type, we did not include MPF in our analysis for this stimulus. Moreover, as observed in the right plot of figure 4, superiority of our model is bolder compared to the simpler stimulus of drifting gratings (right plot of figure 3).

Original: U SING nSPN MODELS AS A MEASURE OF NEURAL CORRELATIONS
Revised: F URTHER ANALYSIS ON THE ROLE OF NEURAL CORRELATIONS

Revised: Without knowledge of the ground truth, data interpretation is much harder especially when the data is limited and high dimensional.

Original: Many experiments have suggested that arousal has the effect of reducing neural correlations, or decreasing the level of synchrony Gandal et al. 
Revised: To further investigate the role of (spatial) neural correlations in population activity we extended our analysis to dynamic latent models that look at each trial as a whole Macke et al. (2011a); Pfau et al. ; Gao et al. Therefore, these models take long temporal correlations into account.

Revised: While these models are usually applied to motor tasks with longer trial duration, temporal correlations could affect the population structure in our tasks as well, especially during the natural movie stimulus.

Revised: We fit a Poisson Linear Dynamical System (PLDS) with nuclear norm penalized rate estimation Pfau et al.

Revised: (2013) 1 to the population activity of primary visual cortex (VISp) during the first and last movie clip in our set (starting from frame 290 and 890), and 0 degree drifting gratings as example extensions of our analysis.

Revised: The number of latent factors were determined by leave-one-out-cross validation over trials (which were 5).

Revised: As shown in the left plot of figure 5, JS-divergence of nSPN were lower for every single stimulus and mouse, especially in the more complex stimuli, i.e. movie clips.

Revised: Notably, latent dynamical systems have demonstrated the ability to capture pairwise correlations, in addition to temporal dynamics of the system (e.g. temporal cross correlations in Macke et al.

Revised: (2011a).

Revised: Therefore, this result suggests that higher order short-term/spatial correlations (20 ms) plays a more crucial role here.

Revised: Using mixtures, nSPN can capture (discretized) temporal dynamics but (probably) not as good as methods with explicit components of modelling dynamics.

Revised: Our second attempt on checking the validity of our conclusions involved previous neuroscience literature.

Revised: Many experiments have suggested that task engagement in the form of attention and arousal has the effect of reducing neural correlations, decreasing the level of synchrony Gandal et al.

Original: ; Uhlhaas et al.
Revised: (2012); Uhlhaas et al.

Revised: (2009), and even measurable behavioral changes Cohen & Maunsell (2009).

Original: These experiments are mostly based on very noisy data such as the Local Field Potential (LFP) and rough estimates of signal frequency Gandal et al.
Revised: These experiments are mostly based on very noisy data such as the Local Field Potential (LFP) and rough estimates of signal frequency Gandal et al.

Original: ; Pfeffer et al.
Revised: ; Pfeffer et al.

Original: ; Vinck et al.
Revised: ; Vinck et al.

Original: Having a quantitatively reliable measure based on spiking data, i.e. ∆ ll SPN , we examined the effect of arousal in the Allen
Revised: Having a quantitatively reliable measure based on spiking data, i.e. ∆ ll SPN , we examined the effect of arousal during drifting grating viewing.

Original: Brain Observatory. Specifically, in each animal we looked at the (ratio to minimum) pupil diameter (a common measure of arousal in mice) changes at each of the chunks described above for all directions that have passed the shuffle test combined.
Revised: Specifically, in each animal we looked at the (ratio to minimum) pupil diameter (a common measure of arousal in mice) changes at each of the chunks described above for all directions. To alleviate the effect of trail-to-trial variability (e.g. caused by behavioral state change) on neural correlations we removed (mouse, direction) pairs with ∆ ll SPN of more than .

Revised: 25 in the primary visual area for the shuffled data.

Original: The mean distribution of Pearson correlation coefficient between pupil diameter and ∆ ll SPN across all animals was significantly below zero for all regions as shown in Fig.
Revised: The mean distribution of Pearson correlation coefficient between pupil diameter and ∆ ll SPN across all animals was significantly below zero for all regions as shown in Fig.

Original: 5 (left).
Revised: 5 (left).

Original: To the best of our knowledge, even with LFP data, there is no work demonstrating the effect of arousal on higher visual areas during an experiment (See Vinck et al.
Revised: To the best of our knowledge, even with LFP data, there is no work demonstrating the effect of arousal on higher visual areas during an experiment (See Vinck et al.

Original: (2015) for VISp).
Revised: (2015) for VISp).

Original: We repeated the same process for MPF by using the difference between log-likelihood of the full (first and second order) model with the first-order only version (second order variables set to zero).
Revised: We repeated the same process for MPF by using the difference between log-likelihood of the full (first and second order) model with the first-order only version (second order variables set to zero).

Original: The result was significantly different (Fig. 5, right).
Revised: The result was significantly different (Fig. 5, right).

Original: Importantly and as mentioned, previous literature results (especially on VISp) are strongly in favour of the SPN results.
Revised: Importantly and as mentioned, previous literature results (especially on VISp) are strongly in favour of the SPN results.

Revised: Importantly, the stimulus in the previous VISp study was also drifting gratings Vinck et al.

Original: Moreover, this result further demonstrates the inability of the PME in modelling the full joint population activity (as opposed to the total population activity).
Revised: Moreover, this result further demonstrates the inability of the PME in modelling the full joint population activity (as opposed to the total population activity).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Correlations are a potentially important channel of information in neural activity, with strong consequences for coding properties.
Revised: Correlations are an important channel of information in neural activity, with strong consequences for coding properties.

Original: Using Sum Product Networks, we have constructed a computationally efficient approach to modeling structure and correlations in populations of spiking neurons.
Revised: Using SPNs, we have constructed a computationally efficient approach to modeling structure and correlations in populations of spiking neurons.

Original: Tested on simulated and experimental data, our approach outperformed both energy-based and latent variable methods.
Revised: Tested on simulated and experimental data, our approach outperformed both energy-based and latent variable methods.

Original: Due to the nature of our scientific question in this paper, i.e. population correlations, we focused on short time spans in which the external variables and animal state remain constant.
Revised: Due to the nature of our scientific question in this paper, i.e. population correlations, we focused on short time spans in which the external variables and animal state remain constant.

Original: Our framework, however, has the capacity to be applied more broadly in computational neuroscience, for example to capture population dynamics in different time scales or as a de-mixing tool for coded features in individual or population of neurons.
Revised: Our framework, however, has the capacity to be applied more broadly in computational neuroscience, for example to capture population dynamics in different time scales or as a de-mixing tool for coded features in individual or population of neurons.

Original: We believe, in these promising future directions, deep and mixed architectures play a larger role.

Original: Reproducibility : The data set used in this paper, Allen Brain Observatory, is publicly available online.
Revised: Reproducibility : The data set used in this paper, Allen Brain Observatory, is publicly available online.

Original: All parameters of the simulated data has been specified in the result section and/or the supplementary material.
Revised: All parameters of the simulated data has been specified in the result section and/or the supplementary material.

Original: Similarly, all parameters related to the methods are mentioned in the results section and/or the supplementary material.
Revised: Similarly, all parameters related to the methods are mentioned in the results section and/or the supplementary material.

Original: Finally, all the code is based on open-source repositories, cited in the results section.
Revised: Finally, all the code is based on open-source repositories, cited in the results section.

Original: Finally, the code for data analysis will be publicly available when the paper gets published.
Revised: Finally, the code for data analysis will be publicly available when the paper gets published.