Paper:
Linear dynamical neural population models through nonlinear embeddings
A body of recent work in modeling neural activity focuses on recovering lowdimensional latent features that capture the statistical structure of large-scale neural populations. Most such approaches have focused on linear generative models, where inference is computationally tractable. Here, we propose fLDS, a general class of nonlinear generative models that permits the firing rate of each neuron to vary as an arbitrary smooth function of a latent, linear dynamical state. This extra flexibility allows the model to capture a richer set of neural variability than a purely linear model, but retains an easily visualizable low-dimensional latent space. To fit this class of non-conjugate models we propose a variational inference scheme, along with a novel approximate posterior capable of capturing rich temporal correlations across time. We show that our techniques permit inference in a wide class of generative models.We also show in application to two neural datasets that, compared to state-of-the-art neural population models, fLDS captures a much larger proportion of neural variability with a small number of latent dimensions, providing superior predictive performance and interpretability.

Review:
proposed novel nonlinear population model and corresponding inference algorithm.
the idea/paper is great, i'm a huge fan. i was, however, confused about a few things. below, i have some general comments, followed by more specifics.   GENERAL  1. accuracy vs. sample size: in statistics, "efficiency" is key (accuracy as a function of sample size).  i want to understand the efficiency for this method for both simulations as well as the real data. ie, please plot error vs sample size for all the methods in each scenario.  2. dimensionality vs. kernel: it is well know that in many cases complexity can be "pushed around", either by keeping more dimensions, or by adding a nonlinear kernel.  i want to understand better how many dimensions the linear model needs to get the accuracy of the nonlinear one.  ie, please plot error vs latent dimensionality for both simulations and real data using both your nonlinear stuff and the linear (inhomogeneous) stuff.  3. baselines: a great feature of simulations is that one can compute bayes optimal performance, to provide an "upper bound".  we can also compute naive performance (eg, a homogeneous poisson model), which provide a null "lower bound". in the real data example, you plotted lower bound.   in both simulation settings, for all experiments, please provide lower and upper bounds.  4. notation: i couldn't quite follow it.   SPECIFICS  1. notation: i found it obtuse, perhaps there is no better way.  but, for example, why is P a function of \ell instead of only \lambda, which is itself a function of \ell?   also, why are \ell and f() separated?  it is because f can be a function of each neuron, and \ell is not?  and if \lambda = \ell, i'm not sure why \lambda exists too.  it was also a bit unclear what role "i" plays: "the i^th element of a deterministic rate function"? but also i \in {1,...,n}.  so i assumed i indexed neuron, so what does it mean to say it is the i^th element of a rate function? it indexed the square bracket, which i guess means that it indexed \ell?  so, i think i would have written:   x_rti | z_rt ~ P( f_i(z_rt))  in any case, please clarify your choices or greatly simplify notation. if the obtuseness is useful for connecting to other stuff,  maybe introduce it as simple as possible first, and then connect it to the other ones?  2. i also didn't understand  "we represent f_\psi through a parametric neural network model" what is a "nonparametric" neural network model, such that you are distinguishing? maybe the word "parametric" in not necessary? please remove or clarify  3.  \theta = (...).  i was a bit confused which of those parameters index a neuron, and which index the whole population.   the dimensionality of \theta was also a bit unclear to me, i realize it is a function of m & n (and the neural network).  but what happened to say, c & d in the PLDS model? are they "eaten" by \psi, meaning you don't even represent them? please clarify which parameters are neuron specific, and the dimensionality of \theta (possibly different dimensionalities for the different special cases).  4.  i believe that PfLDS is a generalization of fLDS, insofar as PfLDS has a nonlinear function of z, and fLDS has a linear one. in other words, the rates are:  exp[ c*z+d ] for fLDS, and exp[ f_\psi(x) ] for PfLDS.  is that correct? and the same kind of thing for GCfLDS? please clarify the text around lines 104-107.   5. "we append the noise model to the acroynm"  i think you mean "prepend"? "append" means put on the end. and acronym is spelled wrong.  6. table 1: i was a bit confused about a few things here.  A) are these numbers good? they are simulations, so you could use the true parameters and estimate the upper bound,  and use some null/dumb model (like constant rate) to provide a lower bound.  i just have no intuition for what PLL or time should be in any of these cases.  B) assuming my comment #4 above is correct, then the fact that PfLDS does nearly identical to fLDS and the same is true for GCfLDS, suggests that f_\psi(.) might have learned a *linear* function (ie, gotten it correct). you don't show it, though you do show something similar for the next simulations.  please show the estimated rate functions (maybe in appendix if you don't have room).  C) you mention the number of trials, neurons, and time bins, as well as the architecture of f_\psi, but not the firing rates or counts.  the estimation error is closely linked to spike count, ie, if your network only fired very sparsely, all estimates would be poor, and your nonlinear estimator would be even worse than the linear ones in many cases.  so, at a minimum, please clarify some summary statistics for the number of spikes per time (maybe per neuron, or something like that).   much better: please conduct an experiment where you compute accuracy as a function of # of spikes in training trials (or some other appropriate measure).  i would expect that the linear estimators outperform the nonlinear ones until spike count is sufficiently high, but i have no intuition whatsoever in terms of how high it must be, in particular under various model misspecification scenarios that you have already described.   the same experiment could be applied to the grid cell simulation.  7. "A paired t-test for PLL is significant (p < 106)."  a t-test?  is PLL really gaussian? i would have thought a wilcoxon test would be much more meaningful.  8. a major claim of the abstract and paper is: "We also show in application to two neural datasets that, compared to state-of-the-art neural population models, fLDS captures a much larger proportion of neural variability with a small number of latent dimensions"  the results you provide indicate that for sufficiently large sample size, upon conditioning on latent dimensionality, your method wins. this is not particularly surprising, because yours is explicitly a nonlinear generalization of the others, so given enough data, it should.  but, this improvement comes at a cost (2 costs). a) it takes more time to fit b) the understanding of each neuron now requires a nonlinear function, rather than a linear one.  a natural question i have therefore is: how does performance change with the number of latent dimensions.  in other words, above i asked for experiments where you conditioned on latent dimension, and change sample size.  i'd also like to see experiments where you condition on sample size, and change the number of latent dimensions.  for example, if adding a single latent dimension to the linear models recovers the additional variance that you obtain from your nonlinear one,  it is not obvious which approach would be preferred.  i suspect that this will not be the case, nonetheless, experimental demonstration would be nice.  9. "cleaner" and "more structured" are words you used to compare the estimated latent states.  these are a bit informal.  i wonder if you can quantify those concepts in a meaningful way?  10. "negative predictive log likelihood" why did you switch from PLL to NLL?  seems like staying with one or the other throughout would be advantageous.  11. "We find 229 that the highly nonlinear generative models used by PfLDS lead to lower-dimensional and hence 230 more interpretable latent-variable representations."  is 1D + nonlinear more interpretable than 2D+linear?   i think this is debatable.  in particular, the highly nonlinear generative models estimated by PfLDS have a highly nonlinear estimate amid the representation, which may or may not help interpretability.  the latent representations are perhaps more interpretable.  this is related to my previous comment.  12. Figure 2: why do (d) and (e) compare to a *homogeneous* Poisson process, rather than PLDS?  i think adding PLDS to this figure would satisfy my desire for understanding the tradeoff between dimensions and nonlinearity in real data (though not in simulation).  13. how should one choose the dimensionality (and neural network architecture) when using your approach? presumably, both depend on sample size.  some guidance would be appreciated.  - jv