Paper: Most discriminative stimuli for functional cell type clustering (ICLR-2024)

Reviewer comment (Reproducibility):
It is stated in the paper "We optimized the stimulus by gradient-ascent, using the SGD optimizer with a learning rate of 100". What are the learnable parameters during this optimization? It is unusual to use such a large value of learning rate.

Author response:
b. The learnable parameters are the pixel values of the maximally discriminable images/video segments, while the network weights are treated as fixed and not being optimized. The high learning rate is typical for pre-image search problems like activity maximization and has been selected as the best one from within the range we explored (10^-3 to 10^3). We will add this information to the paper.

Reviewer comment (Writing):
The statement is ambiguous "After each step we renormalize the video snippet to a L2 norm of 30 and clamp ... into ranges of [-0.913, 6.269] and [-0.654, 6.269]". What does it mean by "a L2 norm of 30"? What are the ranges from?

Author response:
c. The normalization is done because neural responses usually increase with the contrast of the input, but the monitor that displayed the digital twin’s training videos has a finite dynamic range. If we had not constrained the overall energy of the stimuli, the optimized MDS would show an unrealistically high contrast that cannot be displayed by the monitor. This procedure is typical in activity maximization settings (e.g. Walker et al., Nature Neuroscience 2019; Hoefling et al., bioRxiv 2022; Willeke et al, bioRxiv 2023). Even with constraining the L2 norm, single pixels can still exceed the monitor range, which is why we clipped them. The provided intervals correspond to the minimum (UV: -0.913, green: -0.654) and maximum (UV and green: 6.269) light intensity of the monitor, respectively, whereas zero corresponds to the mean luminance across the stimuli. The intervals are asymmetric, because the distribution of light intensities in the stimuli is skewed. We will add this additional information to the paper.

Reviewer comment (Reproducibility):
Where does the digital twin (i.e. the neural network) come from? Is it a pre-trained neural network?

Author response:
1. The digital twin models were trained from scratch on neural responses by authors of previous papers. See answer to a.

Reviewer comment (Experiments):
How are the results/performance sensitive to the hyper-parameters and initializations?

Author response:
2. The three hyperparameters of the method are learning rate, number of initial clusters, and temperature (Eq. 1). The learning rate affects primarily the runtime of the optimization procedure, not the resulting optimal stimuli. We verified that the results are robust with respect to different initializations and number of initial clusters (mouse retina: adjusted rand index ARI=0.87 (page 7) and compare Figure 2A to Appendix Figure 7; marmoset retina: ARI=0.49 (p. 8) and compare Figure 4A to Appendix Figure 8; macaque V4: ARI=0.53 (p. 8) and compare Figure 5 to Appendix Figure 9). The temperature determines how strongly we penalize larger cluster responses. We explored various temperatures in early tests and 1.6 worked well, and we will now run a control where we examine it more systematically and report the results.  

Regarding your question on how our method is sensitive to the temperature hyper-parameter, we now can expand on our previous answer. We systematically swept temperatures logarithmically and ran MDS clustering for three different initializations, and report the median objectives in the table below (for the mouse retina data).

| Temperature |     Objective (Median) | 
|------------:|-----------:|
|   0.2       |  0.67      | 
|   0.4       |  0.89      | 
|   **0.8**   |  **0.98**  | 
|   **1.6**   |  **0.99**  | 
|   **3.2**   |  **0.95**  | 
|   **6.4**   |  **0.92**  | 
|  **12.8**   |  **0.98**  | 
|  25.6       |  0.87      | 
|  51.2       |  0.68      | 


We found a broad, flat objective optimum (bold-face numbers) for temperature values between 0.8 and 12.8 with a consistent number of five to seven resulting clusters. Generally, all clustering algorithms will slightly over- or under-cluster, and to determine a precise number of cell types, functional clusters need to be verified by comparing to morphological, genetical, or transcriptomics data. Analysis in our paper are based on temperature 1.6 with the highest objective value resulting in seven clusters that we successfully matched to an existing hierarchy of cell types (Baden et al. 2016).

For lower/higher temperatures, the objective value and the number of clusters found decreased, indicating worse clustering performance. This results directly from the structure of our softmax objective function, 

$$
    \max_{x_c}J_c=\max_{x_c} \left( \log \frac{\exp \left(\bar r_c(x_c) / \tau \right)}{ \frac{1}{K}\sum_{k=1}^K \exp \left(\bar r_k(x_c) / \tau\right)} \right) \quad \text{(Eq. 1)}\ .
$$

For high temperatures, $\tau \gg \bar r_k(x_c),\ k=1, 2, ..., c, ...$, the inputs to the objective $\bar r_k(x_c) / \tau$ are dominated by the temperature, $\tau$. Hence, the objective becomes insensitive to stimulus dependent inputs $\bar r_k(x_c)$, and clustering performance diminishes. For small temperatures, $\tau \ll \bar r_k(x_c)$, inputs will be strongly amplified and the response of a single cluster $\bar r_{k^*}(x_c)$ will dominate the objective, making it insensitive to the inputs of all other clusters $\bar r_{k \neq k^*}(x_c)$. The resulting MDS will then not suppress all clusters. In conclusion, a good temperature balances these two undesirable regimes.

We will add results of this analysis as a figure and a discussion to the Appendix of the final version of the paper. Thank you very much for asking this interesting question, which will make our paper stronger.  

In summary, the revision we just posted now clarifies your question about details on the digital twin and reports several (new) experiments on the robustness w.r.t. hyperparameters:
1. The procedure is robust against different numbers of initial clusters and the random seed (Section “Robustness” on page 7)
2. The procedure is not very sensitive to the main hyperparameter in the objective: temperature (Appendix Fig. 14)
3. Changing the digital twin architecture does not alter the results substantially (“Robustness” on p. 7 and Appendix Fig. 13)

Reviewer comment (Theory):
The clustering approach is based on the discreteness hypothesis of functional cell types, while ignoring the neurons might be close to the boundary or shows patterns that belongs to two functional cell types in different trials. Therefore, there might be limitation of separability.

Author response:
You are right that discreteness is an underlying assumption. We consider this a feature rather than a problem: the motivation for developing our method was to be able to address the biological question whether neurons form discrete cell types that can be distinguished based on their functional properties. To answer this biological question, we need methods and models that capture certain underlying structural properties. Pitting such models against each other and testing how well they explain a dataset allows us to answer this question. We feel that ICLR as a machine learning venue is the right place for such method contributions and answering the biological question is beyond the scope of this venue.

Reviewer comment (Experiments):
It remains unclear how does the choice of neural networks or back-propagation algorithms affect the solutions.

Author response:
You asked how the choice of neural networks affects our results: We now performed this experiment (Appendix A.6, Fig. 13) and showed that a modified digital twin architecture does not affect the results much. The resulting clusters and MDS look similar to those obtained with our initial architecture. Relatedly, we also addressed the robustness w.r.t. hyperparameters (Paragraph “Robustness” on p.7).

Reviewer comment (Evaluation):
Qualitatively, it is difficult to compare and interpret different MDS, and understand how they diverge from each other.

Author response:
You are right that it might be difficult to compare and interpret the MDS. However, keep in mind that (a) in practice one would rarely use the MDS alone to characterize a neuron and (b) the MDS are still useful even if they are not interpretable at all, because experimentalists can use them to quickly determine a neuron’s type based on a small set of stimuli.

Reviewer comment (Experiments):
Simple baselines like searching nearest neighbor in original image space, or simple image interpolation could be introduced and compared.

Author response:
Regarding your suggestion about a baseline in image space: We have now hypothesized about what you may have had in mind. One could replace the optimization of a maximally discriminative stimulus with a search for a maximally discriminative video snippet in our dataset for which we have measured neural responses. Unfortunately, such an approach would not be promising, because the video snippet from the mouse retina dataset (Höfling et al. bioRxiv 2022) most similar to a MDS had a cosine similarity of 0.4, for all other snippets it was below 0.2. This observation suggests that a search in image space for dataset examples is not very promising, because the dataset is too small. Using a larger dataset would not be useful either, because this would again require a predictive model and an exhaustive search through the dataset, which is likely to be more expensive than an optimization-based pre-image search as we perform it.

Reviewer comment (Experiments):
It seems that the optimized MDS has not been tested on real mouse yet, only evaluated on holdout neurons and trials.

Author response:
You mentioned the importance of in-vivo evaluation. We now have added text to the Discussion, clearly stating that ultimately the method needs to be tested and validated in-vivo. In addition, we now mention that both the single cell MEIs for the digital twins and optimizer we used were successfully validated in-vivo (Höfling et al. bioRxiv 2022 and Willeke et al. bioRxiv 2023), suggesting our results might well generalize to an in-vivo experiment, too.

Reviewer comment (Writing):
How does functional cell types here related to neuron classes of excitatory and inhibitory?

Author response:
Whether and how these clusters relate to excitatory vs. inhibitory cell types is an interesting question, but cannot be addressed with these datasets. The mouse retina dataset contains some displaced amacrine cells, but we restricted our analysis to the most widely available cell types in the dataset, which were all ganglion cells and, hence, excitatory. In the marmoset retina, we expect only ganglion cells. In the monkey V4 dataset we do not have information on cell types as they were recorded with extracellular multi-electrode recordings. We will revise the dataset description to mention this, accordingly.

Reviewer comment (Evaluation):
How to explain and interpret some off-diagonal entries in Fig 2D and Fig 4A?

Author response:
Regarding your question on the interpretation of off-diagonal entries in Fig. 2: in our previous reply we mentioned two explanations; one of them is the limitation that some cell types may not be identifiable by a single image/video MDS (e.g. ON-OFF cells). We now extended our method to optimize two maximally discriminative stimuli instead of one, which revealed an additional cluster mainly associated with ON-OFF cells (Appendix A.5 and Fig. 12). This proof of principle shows that there is a path towards even more refined clustering based on our overall setup.

Reviewer comment (Theory):
A potential weakness in the presented approach is that it assumes that the most informative stimuli for classifying cell types can be automatically chosen without requiring domain knowledge or expert input. While this is presented as an advantage, it may also be a limitation, as there could be cases where domain-specific insights are necessary for more accurate and nuanced cell type classification.

Author response:
2. Domain specific knowledge: You are completely right - leveraging domain knowledge is of course helpful if it exists. In later processing stages such as V4, however, we simply do not know that much. In such situations we believe our approach is advantageous because it is still applicable. It is important to realize that one important use case of our method is to facilitate scientific discovery, i.e. specifically situations where the domain knowledge does not yet exist. We will improve the paper to more clearly discuss this.

Reviewer comment (Experiments):
The success of the approach relies on the availability of a "digital twin" dataset, which may not always be readily available for all experimental studies, potentially limiting its applicability.

Author response:
3. Regarding the availability of digital twin models: Please note that the digital twin is needed only to find out what the cell types are. An important use case of our method is to quickly assign neurons to already identified types. In this situation, only the MDS are needed, but no digital twin. We will revise the paper to mention this explicitly.

Reviewer comment (Experiments):
While the authors express their belief in the usefulness of the algorithm and stimuli, there is no mention of empirical results or validation studies that demonstrate the actual effectiveness and applicability of the proposed approach.

Author response:
You point out the absence of empirical results (first question) and a benchmark (third question) as significant concerns, but we are unsure if we understand your suggestion correctly: For mouse retinal ganglion cells, Baden et al. (2016) established functional types based on a variety of functional and morphological properties. We tested our algorithm on a dataset labeled with those types and demonstrated good agreement of our clustering algorithm to the Baden et al. (2016) cell types (see first part of the Results and Figure 2). These results were empirically obtained on real retina responses and, in our opinion, represent exactly the benchmark you were asking for. Hence, we are unsure if we understood your main concern correctly. Could you clarify if this resolves your concern and what you suggest we do to increase the score you rated our paper with?

Reviewer comment (Experiments):
One potential weakness of the clustering algorithm is its limitation in effectively distinguishing certain cell types with complex responses. For instance, for ON-OFF RGCs that respond to both light increments and decrements, optimizing a single, short MDS to maximize their response may not adequately capture their unique properties. Similarly, for cell types like "OFF suppressed 2," which exhibit a high baseline firing rate mostly suppressed by stimulation, the algorithm may not effectively identify them without specific adaptations, potentially requiring additional stimulus optimization strategies for better classification.

Author response:
1. Limitations: We agree that our algorithm is limited in distinguishing certain types with complex responses, e.g. ON-OFF cells from ON or OFF cells, which is why we discuss it explicitly in the discussion. Every method has limitations, including alternative methods that address similar problems (Ustyuzhaninov et al., bioRxiv 2022; Willeke et al., bioRxiv 2023). Note that ours is the first that has actually been validated on empirical data where something akin to biological ground truth exists (Baden et al. 2016). Beyond the retina (i.e. in visual cortex), we simply do not know whether functional types exist and, if so, how to identify them. Closing this gap is the key motivation behind our paper and, hence, an important contribution even if it has limitations. For sure, some of these limitations can be addressed, e.g. by extending our algorithm to generate multiple stimuli per cluster. Such improvements are interesting avenues for future work but beyond the scope of the current paper.  

In your initial review and your response, you were interested in how our method could be extended to distinguish ON-OFF cells from ON or OFF cells by optimizing multiple MDS at the same time - an exciting idea we now would like to follow up on. We extended our method to optimize two maximally discriminative stimuli instead of one, which revealed an additional cluster mainly associated with ON-OFF cells (Appendix A.5 and Fig. 12). This proof of principle shows that there is a path towards even more refined clustering based on our overall setup.

Reviewer comment (Experiments):
The absence of a benchmark is a significant weakness in this paper. It would be helpful if the author can add any performance to determine its effectiveness or limitations.

Author response:
You point out the absence of empirical results (first question) and a benchmark (third question) as significant concerns, but we are unsure if we understand your suggestion correctly: For mouse retinal ganglion cells, Baden et al. (2016) established functional types based on a variety of functional and morphological properties. We tested our algorithm on a dataset labeled with those types and demonstrated good agreement of our clustering algorithm to the Baden et al. (2016) cell types (see first part of the Results and Figure 2). These results were empirically obtained on real retina responses and, in our opinion, represent exactly the benchmark you were asking for. Hence, we are unsure if we understood your main concern correctly. Could you clarify if this resolves your concern and what you suggest we do to increase the score you rated our paper with?

Reviewer comment (Evaluation):
My primary concern is how useful and interpretable the method will be for actual practice. The experiments essentially treated a carefully-examined existing publication as the ground truth to compare with. I assume in a lot of real scenarios, we might already have this kind of biological baselines. For cases where they are not available, the interpretability of the identified MDS might be important.

Author response:
Thank you for your encouraging feedback on our paper. We would like to start by addressing your main concerns about usefulness in practice and interpretability of the method. Please let us know if this addresses your concern or what you believe we should do or demonstrate to increase your score.

Regarding usefulness: As you state correctly, we used the cell types by Baden et al. (2016) to show our algorithm’s effectiveness. However, your assumption that the same biological baselines exist in a lot of real scenarios is not true. Beyond the retina (i.e. in visual cortex), it is not known if cells can be clustered into functional types and, we do not know how many types exist. Additionally, it is not clear how exactly to identify them because the method used by Baden et al. (2016) does not generalize to visual cortex. Closing this gap is the key motivation behind our paper: Our clustering approach finds optimal stimuli to determine functional types and assigns neurons quickly to their type.

Regarding interpretability: Currently there exist no experimentally validated methods. The methods that do exist use either in-silico experiments with simple, classical stimuli (Ustyuzhaninov et al., bioRxiv 2022; but this approach does not generalize to lesser known areas such as V4) or similarity of single cell maximally exciting images (Willeke et al, bioRxiv 2023). In contrast to these approaches, our method offers the added value that the stimuli we obtain are not only highly activating, but actually visually what feature a cell type “specializes” on that other cell types do not respond to. Thus, our method offers potential to drive the field forward exploring functional cell types for many real world scenarios where no clustering baselines exist (e.g. Ustyuzhaninov et al., bioRxiv 2022; Willeke et al, bioRxiv 2023).

We will revise the paper to discuss these points more clearly.

Reviewer comment (Presentation):
Can the authors elaborate more on Figure 2B? They referenced figure 2B when stating in the text that “none of the MDS exhibited direction selectivity”, but I didn’t quite understand why Figure 2B is referenced here.

Author response:
2. We agree, in the first principle component (Figure 2B) direction selectivity cannot be examined. The digital twin model (Höfling et al. bioRxiv 2022) used space-time separated convolution learning a time-trace and spatial filter independently (similar as depicted in Figure 2B), and hence was not able to learn direction selectivity. We apologize for referencing Figure 2B for such claims and will revise the paper accordingly.

Reviewer comment (Experiments):
For the experiments in Figure 2, the identified MDS seems to be largely combination of several functional cell types from the published Baden et al (2016) paper. I wonder if further tweaking with the sub-clustering procedure can better recover the published functional cell types.

Author response:
Baden et al. (2016) relied on functional responses to chirp and moving bar stimuli and morphological properties. As the digital twin cannot model direction selectivity or morphological properties, we removed these features from the classifier that was used to label our dataset, showing degraded separability across the types reported in Baden et al. (2016) (Figure 2E). Hence, the digital twin lacking these features poses a limit on the granularity of our clusters. 

We added another experiment that pertains to your third question whether further tweaking could better recover the functional cell types published by Baden et al. (2016): We extended our method to optimize two maximally discriminative stimuli instead of one, which revealed an additional cluster mainly associated with ON-OFF cells (Appendix A.5 and Fig. 12). This proof of principle shows that there is a path towards even more refined clustering based on our overall setup.

Reviewer comment (Experiments):
How will the result change if the number of gradient ascent steps is changed during each M-step? Trying to understand if there is a delicate balance needed between how well the M-step is optimized and the E-step, or if the procedure is relatively robust and almost guaranteed to converge to the same set of results.

Author response:
4. In the M-step we optimize MDS until the objective does not change significantly anymore to yield stable evaluations of our objective (Eq. 1). During algorithm development for the mouse retina data we verified sufficient training steps were performed by inspecting the objective values across steps. For marmoset and monkey V4 data, we automatically checked for convergence: we stopped MDS optimization if an exponential moving average of the objective did not increase anymore for 100 optimization steps and verified from the objective values across optimization steps that this stopping criterion led to a reasonably converged objective. We now have realized this information was not clearly mentioned throughout the paper, which we will revise accordingly.