Paper: SiBBlInGS: Similarity-driven Building Block Inference using Graphs across States (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the paper mainly comes from borrowing the idea and success of functional Building Blocks (BB) into the neural data modeling.

Author response:
The novelty of our work goes far beyond the application of a concept to neuroscience. 
It lies in the development of a versatile framework that identifies functional building blocks through re-weighted temporal graphs, fostering sparsity in multi-way data while considering within-state and between-state information.

Our method tackles issues of real world data that other models typically overlook, such as varying trial durations, different trial counts per state, and handling of missing data or sampling rates, as well as enabling the discrimination between background and state-specific building blocks. 

The potential applications of this method extend well beyond neuroscience, with potential applications in genomics, consumer behavior, financial markets, environmental sciences, urban planning, and the study of social dynamics 
(For additional details, please refer to the examples provided in our second response to reviewer T8G6). 
We believe our approach is a standalone method with universal applicability, including contributions to neuroscience, as well as to other fields.

Reviewer comment (Theory):
With a stacking of state-of-the-art techniques and domain knowledge, the proposed method achieves effectiveness through empirical evidence. However, these findings are a bit heuristic and empirical.

Author response:
Our paper presents a comprehensive empirical exploration of SiBBlInGS, highlighting its adaptability and broad applicability through various synthetic and real-world examples. The approach is grounded in the theoretical principles of dictionary learning, drawing from seminal works in the field (e.g., [1], [2], [3], [4]). However, we acknowledge that a comprehensive theoretical discourse on dictionary learning is beyond the scope of our 9-page, application-focused paper.

In particular, in [1], the authors provide a geometric characterization of the optimization landscape in dictionary learning, eliminating concerns over spurious local minimizers. [2] shows theoretical guarantees that detail how searching within an over-realized model space can foster the recovery of dictionaries, conceptually linking this process with generalization bounds and introducing an efficient strategy for extracting accurate model components. [3] establishes the theoretical framework necessary for confidently recovering dictionaries under sparsity constraints, and [4] marks a deep dive into sample complexity, stating how much data is needed to reliably engage in dictionary learning and parallel matrix factorization efforts.

We will include these references with a brief explanation in the introduction of the updated paper to emphasize the above.


[1] Sun, J., Qu, Q., & Wright, J. (2016). Complete dictionary recovery over the sphere I: Overview and the geometric picture. IEEE Transactions on Information Theory, 63(2), 853-884.

[2] Sulam, J., You, C., & Zhu, Z. (2022). Recovery and generalization in over-realized dictionary learning. The Journal of Machine Learning Research, 23(1), 6060-6082.

[3] Gribonval, R., & Schnass, K. (2010). Dictionary Identification—Sparse Matrix-Factorization via $\ell_1 $-Minimization. IEEE Transactions on Information Theory, 56(7), 3523-3539.

[4] Gribonval, R., Jenatton, R., Bach, F., Kleinsteuber, M., & Seibert, M. (2015). Sample complexity of dictionary learning and other matrix factorizations. IEEE Transactions on Information Theory, 61(6), 3469-3486.

Reviewer comment (Theory):
There are few theoretical guarantees in this paper.

Author response:
Our paper presents a comprehensive empirical exploration of SiBBlInGS, highlighting its adaptability and broad applicability through various synthetic and real-world examples. The approach is grounded in the theoretical principles of dictionary learning, drawing from seminal works in the field (e.g., [1], [2], [3], [4]). However, we acknowledge that a comprehensive theoretical discourse on dictionary learning is beyond the scope of our 9-page, application-focused paper.

In particular, in [1], the authors provide a geometric characterization of the optimization landscape in dictionary learning, eliminating concerns over spurious local minimizers. [2] shows theoretical guarantees that detail how searching within an over-realized model space can foster the recovery of dictionaries, conceptually linking this process with generalization bounds and introducing an efficient strategy for extracting accurate model components. [3] establishes the theoretical framework necessary for confidently recovering dictionaries under sparsity constraints, and [4] marks a deep dive into sample complexity, stating how much data is needed to reliably engage in dictionary learning and parallel matrix factorization efforts.

We will include these references with a brief explanation in the introduction of the updated paper to emphasize the above.


[1] Sun, J., Qu, Q., & Wright, J. (2016). Complete dictionary recovery over the sphere I: Overview and the geometric picture. IEEE Transactions on Information Theory, 63(2), 853-884.

[2] Sulam, J., You, C., & Zhu, Z. (2022). Recovery and generalization in over-realized dictionary learning. The Journal of Machine Learning Research, 23(1), 6060-6082.

[3] Gribonval, R., & Schnass, K. (2010). Dictionary Identification—Sparse Matrix-Factorization via $\ell_1 $-Minimization. IEEE Transactions on Information Theory, 56(7), 3523-3539.

[4] Gribonval, R., Jenatton, R., Bach, F., Kleinsteuber, M., & Seibert, M. (2015). Sample complexity of dictionary learning and other matrix factorizations. IEEE Transactions on Information Theory, 61(6), 3469-3486.

Reviewer comment (Theory):
How does the proposed method ensures to model and distinguish the within-state and between-state variabilities and relationships, which could be crucial for a comprehensive analysis of the data.

Author response:
**Distinguishing Within and Between State Variability:**

SiBBlInGS  adeptly integrates analysis both within and across states, maintaining consistency of the structure of within state BBs and accommodating nuanced controlled variations of between-state BBs. The degree of resemblance between corresponding cross-state BBs is methodically adjusted to reflect the corresponding levels of similarity between the states (as seen in the last term of Equation 1).

Reviewer comment (Theory):
What are the bio-plausible insights of this paper and in the model design?

Author response:
We thank the reviewer for the comment on bio-plausible insights and model design.

**Bio-plausible model design:**

SiBBlInGS is informed by biologically plausible principles and is designed to uncover functional groups within real-world data, including biological data, based on several key features that align with biology:
- *Temporal Similarity-Based Functional Groups:* 

SiBBlInGS identifies functional groups rooted in temporal similarity, reflecting how some real-world entities like neurons 
 functionally co-activate (e.g., as shown in [1,2,3]).

- *Beyond Orthogonality Assumption:*

The model challenges traditional orthogonal assumptions of components made, for instance, by PCA, HOSVD, and SRM [4], acknowledging that biological building blocks, like functional neural ensembles, may exhibit non-orthogonal interactions.

- *Sparsity in Representations:* 

SiBBlInGS integrates sparsity, recognizing that in biological systems like the brain, functionally related groups are typically sparse subsets of the whole simultaneously recorded units.

- *Handling States & Trials:* 

The model naturally handles data with multiple trials and several states, which is a common structure in real-world biological datasets. This is crucial for delineating variability across different conditions in scientific research, such as varying disease stages or comparisons between disease and control groups.

- *Variable Trial Durations:*

The model accommodates trials with variable durations, aligning with the variability seen in real time series data where the duration of events, like animal behavior tasks, is not necessarily fixed.

- *Missing Data and  Sampling Rates:* 

SiBBlInGS is built to work with missing data points and varying sampling rates, addressing issues common in datasets ranging from clinical visits to temporary lapses in sensor measurements.


**Bio-plausible Insights:**

In our paper, the emphasis was on showcasing the effectiveness of SiBBlInGS across various data types and highlighting its potential for advancing scientific discoveries, rather than being mainly focused on the biological insights.

We do demonstrate though the model's potential in finding interpretable meaningful neural ensembles and in detecting subtle cross-state variations in these ensembles across states, along with informative traces. Specifically:
- *Functional Neuronal Groups in Motor Tasks:*

For recordings from area 2 of the cortex, SiBBlInGS successfully discerned several neuronal groups engaged during a reaching task. The model found subtle adaptations of the cross-state neural BBs’ structures (Figure 4B), and showed that their temporal signatures can encode  the monkey’s decision-making process (Figure 4C). 

- *Seizure Detection in EEG Data:* 

The model proved its efficacy in an EEG data from an epileptic patient, where, unlike other methods, it identified the seizure onset location seconds prior to the event. This indicates SiBBlInGS's promise in epilepsy research.  



These results highlight the model's versatility and scientific utility, suggesting its potential for valuable insights in biological and  scientific research. We anticipate employing it in upcoming scientific projects for biological discoveries.

[1] Gerstein, G. L., Perkel, D. H., & Subramanian, K. N. (1978). Identification of functionally related neural assemblies. Brain research, 140(1), 43-62.

[2] Malik, A. N., Vierbuchen, T., Hemberg, M., Rubin, A. A., Ling, E., Couch, C. H., ... & Greenberg, M. E. (2014). Genome-wide identification and characterization of functional neuronal activity–dependent enhancers. Nature neuroscience, 17(10), 1330-1339.

[3] Yuste, R., Nelson, D. A., Rubin, W. W., & Katz, L. C. (1995). Neuronal domains in developing neocortex: mechanisms of coactivation. Neuron, 14(1), 7-17

[4] Chen, P. H. C., Chen, J., Yeshurun, Y., Hasson, U., Haxby, J., & Ramadge, P. J. (2015). A reduced-dimension fMRI shared response model. Advances in neural information processing systems, 28.

Reviewer comment (Experiments):
The authors compare their model only to very simple baselines, such as PCA and vanilla PARAFAC. In particular, in the context of PARAFAC a lot of recent literature exists that generalized PARAFAC to explicitly account for temporal dependencies. Such approaches should be discussed explicitly and systematically benchmarked. In particular, the authors should consider [1,2,3], where temporal information and different states over time are modelled via GP priors or parametric regularizers.

Author response:
We appreciate the reviewer's thoughtful feedback on our manuscript and the referral to additional sources and references.

**Additional Baselines:**

We appreciate the feedback from the reviewer. In response, we have provided a brief description of additional models in the ''BACKGROUND AND RELATED WORK" (see in blue font) to underscore the distinctions from SiBBlInGS.

Furthermore, we added two more experiments, comparing our method also to: 1) NONFAT [1], and 2) NNDTN (discrete-time NN decomposition with nonlinear dynamics). The modifications include updates to Figure 2 in the paper to incorporate these comparisons, along with the addition of Figures 14 and 15, illustrating the identified components for all methods and the correlation with the ground truth per BB and per state. Please consult the updated paper for details on how we extracted these components, as described in the supplementary material. 

Broadly, SiBBlInGS serves a distinct purpose compared to methods mentioned in [1, 2, 3] or other Gaussian Process (GP)-based approaches. While these methods leverage temporal dependencies for robust decompositions with predictive power for future time points, SiBBlInGS aims to identify interpretable functional groups based on co-activity with subtle controlled variability along different axes—the trial and state axes, which are orthogonal to the time axis. 
In particular, SiBBlInGS focuses on discovering interpretable components with their traces rather than predicting future time points (as presented in [1] for instance) or solely reconstructing test data. 

Furthermore, the identified components in some of these other methods are not naturally sparse, and they do not aim to group sparse subsets of channels based on per-state temporal similarity. Hence, their interpretation does not align well with the discovery of functional groups that, in various fields, are assumed to represent co-active components, such as neural ensembles. Moreover, some of these methods cannot naturally work with data of varied duration or rates due to the inherent tensor structure. 

Although integrating GP ideas and dynamical processes into the analysis, as seen in [1, 2], is promising and could be very interesting for future work to better model within-trial non-stationary temporal conditions, it is not the primary goal of the current framework.

[1] Wang, Z., & Zhe, S. (2022, June). Nonparametric Factor Trajectory Learning for Dynamic Tensor Decomposition. In International Conference on Machine Learning (pp. 23459-23469). PMLR.

[2] Tillinghast, C., Fang, S., Zhang, K., & Zhe, S. (2020, November). Probabilistic neural-kernel tensor decomposition. In 2020 IEEE International Conference on Data Mining (ICDM) (pp. 531-540). IEEE.

[3] Ahn, D., Jang, J. G., & Kang, U. (2021, October). Time-aware tensor decomposition for sparse tensors. In 2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA) (pp. 1-2). IEEE.

Reviewer comment (Experiments):
The experiments are very limited. While there are some analyses on synthetic data, they should be extended to include more baselines and also demonstrate how the proposed Method works in different settings for high-dimensional time series data, e.g., such as analysed in [1].

Author response:
We thank the reviewer for the comments regarding the experiments and presentation.

**Experiments:**

With respect to the concern about the experiment's scope, we've strived to maximize the presentation of our model's efficiency within the 9-page constraint for ICLR papers. The paper includes four experiments, each exploring different settings, characteristics, and conclusions. In response to the feedback and within the short rebuttal period, and as you suggested, we've added two more experiments, specifically comparing SiBBlInGS to NONFAT [1] and to NNDTN (discrete-time NN decomposition with nonlinear dynamics, a model mentioned in [1] as well). Yet, it is crucial to underscore the differences in both goal and assessment methodology between SiBBlInGS and the approach and the baselines presented in [1]. The objective of [1] and most of its baselines revolve around considering temporal non-stationarities through Gaussian process priors utilizing the frequency domain. In contrast, SiBBlInGS is designed to discover interpretable sparse BBs that represent functional groups based on similar activity, and seeks to understand how these groups vary across and within conditions, with changes occurring over trials—a different axis of focus compared to [1]. Moreover, as demonstrated in the cortex experiment, due to regularizations on $\Phi$—giving the model a high number of BBs as an hyperparameter (i.e. too large $p$) will result in non-active or noise-capturing BBs (Figure 4).

Regarding the discussion of baseline results in real-world applications, we recognize the importance of comparisons. However, as the BBs of these real-world examples are unknown, and it is precisely what SiBBlInGS aims to reveal, quantitative comparisons  are intractable (given the unsupervised nature). Our approach has been to showcase SiBBlInGS' capability in recovering ground truth BBs in synthetic data, and then demonstrate its effectiveness  for the real-world examples (where ground truth BBs are unavailable) in providing interpretable, meaningful, sparse BBs with cross-state informative controlled variations.  

Additionally, for the EEG experiment, we present results from other baselines in Figure 13, where it is qualitatively evident that these models fall short in capturing the structural emergence of a BB around the area the seizure originates from. Unfortunately, for that experiment, a comparison with NONFAT and NNTDN is unfeasible, as these methods are based on Torch and are computationally expensive, rendering them impractical to run on our PCs during the rebuttal for this high-dimensional data—this, in itself, underscores another advantage of SiBBlInGS.

Reviewer comment (Evaluation):
Results from baselines should also be discussed for the real-world applications.

Author response:
We thank the reviewer for the comments regarding the experiments and presentation.

**Experiments:**

With respect to the concern about the experiment's scope, we've strived to maximize the presentation of our model's efficiency within the 9-page constraint for ICLR papers. The paper includes four experiments, each exploring different settings, characteristics, and conclusions. In response to the feedback and within the short rebuttal period, and as you suggested, we've added two more experiments, specifically comparing SiBBlInGS to NONFAT [1] and to NNDTN (discrete-time NN decomposition with nonlinear dynamics, a model mentioned in [1] as well). Yet, it is crucial to underscore the differences in both goal and assessment methodology between SiBBlInGS and the approach and the baselines presented in [1]. The objective of [1] and most of its baselines revolve around considering temporal non-stationarities through Gaussian process priors utilizing the frequency domain. In contrast, SiBBlInGS is designed to discover interpretable sparse BBs that represent functional groups based on similar activity, and seeks to understand how these groups vary across and within conditions, with changes occurring over trials—a different axis of focus compared to [1]. Moreover, as demonstrated in the cortex experiment, due to regularizations on $\Phi$—giving the model a high number of BBs as an hyperparameter (i.e. too large $p$) will result in non-active or noise-capturing BBs (Figure 4).

Regarding the discussion of baseline results in real-world applications, we recognize the importance of comparisons. However, as the BBs of these real-world examples are unknown, and it is precisely what SiBBlInGS aims to reveal, quantitative comparisons  are intractable (given the unsupervised nature). Our approach has been to showcase SiBBlInGS' capability in recovering ground truth BBs in synthetic data, and then demonstrate its effectiveness  for the real-world examples (where ground truth BBs are unavailable) in providing interpretable, meaningful, sparse BBs with cross-state informative controlled variations.  

Additionally, for the EEG experiment, we present results from other baselines in Figure 13, where it is qualitatively evident that these models fall short in capturing the structural emergence of a BB around the area the seizure originates from. Unfortunately, for that experiment, a comparison with NONFAT and NNTDN is unfeasible, as these methods are based on Torch and are computationally expensive, rendering them impractical to run on our PCs during the rebuttal for this high-dimensional data—this, in itself, underscores another advantage of SiBBlInGS.

Reviewer comment (Presentation):
I find the presentation of the paper could be improved: it requires a lot of jumping back and forth between appendix and main text for important results and to gain a good understanding. I also found the results of the real-world applications hard to understand.

Author response:
We thank the reviewer for the comments regarding the experiments and presentation.

**Presentation:**

Thank you for your feedback on the results presentation. We acknowledge the constraints of conference papers, which limit us to 9 pages. In response, we've prioritized clarity in the model description and highlighted various experiments showcasing different features. Supplementary details, considered less urgent, have been included for completeness. In light of your comment, we've considered enhancing the presentation flow by first introducing the inference for $A$ and only then presenting the kernels. Please find this modification in the attached updated paper.

Reviewer comment (Reproducibility):
The proposed framework is complicated. While the paper provides a high-level overview of the SiBBlInGS framework, it does not provide detailed implementation instructions or code. This may make it difficult for researchers to replicate the results or apply the framework to their own data.

Author response:
**Implementation Instructions and Code:**
As presented in Algorithm 1 (page 14) as well as given the code that was attached in the supplementary materials, we believe the algorithm can be used by future researchers as well. Once accepted and dis-anonymized, a GitHub repository with the code will be shared and a Python PyPi package for simple pip installation will be created for simple usage of the algorithm and to enable users to develop custom implementations and modifications.

Reviewer comment (Experiments):
While the paper discusses potential applications of the SiBBlInGS framework in scientific domains, it does not provide concrete examples of real-world applications. This may limit the impact of the framework and its adoption by researchers in different fields.

Author response:
**Examples for Real World Applications:**
In addition to the real-world examples mentioned in the ''CONCLUSIONS AND DISCUSSION" section (section 6), please find below some more details about potential applications of SiBBlInGS:
- *Consumer Products:* SiBBlInGS could identify groups of products (channels) with similar purchasing patterns over time (time) across 
countries (states), and explore how these patterns vary by location to inform personalized customer recommendations.
- *Economic Analysis:* SiBBlInGS could be applied to cluster stock volume data (channels) over time (time) and across countries (states), to uncover patterns reflecting cross-country market trends and differences in investor activity.
- *Climate Science:* SiBBlInGS could be used to cluster carbon dioxide emission patterns of various countries (channels) across daily intervals (time) and yearly changes (states). This analysis aids in discerning global emission trends and could inform more targeted climate policy initiatives.
- *Social Dynamics Analysis:* SiBBlInGS could be utilized to analyze and understand the formation and evolution of online user groups (channels’ BBs) based on shared temporal patterns of user activity (time) across different states (e.g. before/ during / after the COVID-19 pandemic). This approach could enable the observation of shifts in social interactions and group dynamics in response to global events.
- *Genetics Research:* SiBBlInGS could analyze multi-tissue gene expression data (e.g., as the data described in [1]) to identify functional gene groups (channels) and assess how their expression across tissues (~time) differs between individuals (states).

[1] Hore, V., Vinuela, et al. (2016). Tensor decomposition for multiple-tissue gene expression experiments. Nature genetics, 48(9), 1094-1100.

Reviewer comment (Experiments):
Can the authors discuss the potential limitations of the SiBBlInGS framework in terms of scalability and computational efficiency? For example, how might the framework perform on larger datasets or in real-time applications, and what steps can be taken to address these limitations?

Author response:
**Potential limitations and improvements of SiBBlInGS in terms of scalability and computational efficiency:**
As shown in Section E on model complexity, the current implementation's speed scales with the dataset's dimensionality, encompassing the number of channels, states, time points, and the rank of each trial. To enhance computational speed, the implementation already incorporates random subsampling of time periods and trials within each state during iterations (as mentioned in the paper after equation 1). This allows the model to process only a portion of the data at each update step, eventually exposing the model to all trials and time points. Additionally, the current implementation offers users to choose between different lasso solvers that vary in speed. Future improvements for efficiency may involve implementing the model in languages other than Python, such as C++ or Julia, and applying parallel computing for faster and more efficient performance.
With respect to real-time applications, SiBBlInGS is not intended for real-time inference. However, if BBs are found previously based on some training data for a state, they can be easily used in real-time applications to find their respective traces when new observations arrive (i.e., by solving equation 9).

Reviewer comment (Experiments):
How does the proposed method compare to other approaches for discovering fundamental representational units within multi-dimensional data?

Author response:
**Comparing SiBBlInGS with Other Approaches:**

As demonstrated in the synthetic data results (Figure 2), EEG experiments (Figure 13), and trends experiments (Figure 8), along with additional comparisons detailed in Table 2, we assessed our method against alternative tensor and matrix factorization techniques, through both theoretical analysis and practical experiments.

SiBBlInGS stands apart from classic tensor factorization methods, as it finds similarity-driven sparse BBs with subtle cross-state variations, accommodates variable trial durations and inherently incorporates state information and similarities into the analysis, effectively differentiating intra- and inter-state variability— aspects not addressed by other approaches. Additionally, certain methods presume orthogonality among components, a condition that may not hold true for some real-world cases (e.g.,  neural ensembles). These alternative approaches often tend to find components based on maximum variance or energy rather than co-activity, and they are not inherently tailored to differentiate state-specific from state-invariant building blocks (as $\nu$ does in SiBBlInGS).

Reviewer comment (Theory):
How does the shared temporal activity, inter- and intra-state relationships, and non-orthogonal components contribute to the discovery of building blocks and the effectiveness of the framework?

Author response:
**Significance of Temporal Activity, Inter/Intra-State Relationships, and Non-Orthogonality:**

*Shared Temporal Activity:*

We appreciate the reviewer’s question about the contributions of different features of SiBBlInGS to the framework.
As detailed following Equation 1 in the paper (starting with 'The weighted evolving regularization terms for each state...'), the channel similarity graph (${H}$) is built based on temporal activity similarities between channels per trial. This graph is integrated into the denominator of the regularization weight ($\lambda$) in the re-weighted lasso equation (2nd part of Equation 1) and thus fosters the aggregation of channels with co-active patterns into the same BBs. 

In particular, the term 
$H_{n:}A_{:j}$ 
that appears in the denominator for updating $\lambda_{n,j}^d$ (2nd formula in Equation 1) reflects the correlation between the activity of the $j$-th BB (as defined by $A_{:j}$) with the similarity levels of all channels to the $n$-th channel (as defined by $H_{n:}$). 
Since that term appears in the denominator of $\lambda_{n,j}^d$—a higher correlation between $A_{:j}$ and $H_{n:}$ (which implies that the temporal neighbors of the $n$-th channel might be in the $j^{th}$ BB), results in a smaller sparsity weight ($\lambda_{n,j}^d$ decreases)—thus promoting that $n^{th}$  channel's inclusion in the $j^{th}$ BB and effectively ‘’pushing” the  channel to the BB that includes its temporal neighbors.

Conversely, if the temporal neighbors of channel $n$ mainly do not belong to that $j$-th BB, the term  $H_{n:}A_{:j}$ will be smaller, hence $\lambda_{n,j}^d$ is larger, resulting in an increased sparsity regularization that effectively pushing that $n$-th channel out of the $j^{th}$ BB. 

Another effect of the shared temporal activity is expressed in the data-driven approach, as the state graph is a function of the temporal similarities between states, and the regularization on the distance between cross-state BBs is proportional to the temporal similarities between these states (stored by $P$, and as seen in Equation 1), thereby encapsulating state-temporal similarities within the analysis. 

Specifically, last term in Equation 1 ($P_{dd'} \|\|a_{i:} - a_{j:}\|\|_2^2$) modulates the $\ell_2$ norm on the distance between corresponding BBs across different states in accordance with the states' similarities, ensuring that more similar states are associated with more similar BB patterns.



*Non-orthogonal components:*

Our method's ability to identify components without enforcing orthogonality constraints, as opposed to techniques like HOSVD or PCA, aligns better with real-world data where such constraints may not naturally exist. For example, in the context of neural ensembles, it is plausible to encounter functional groups that are non-orthogonal, both structurally and temporally.



*Intra vs inter state relationships:*

As written just before Equation 1 (the paragraph starting with “Assuming consistent underlying groups between same-state trials,....”), the BBs are kept constant between trials of the same state while supporting subtle controlled structural variations between states, thus addressing both resolutions. Unlike the regularized BBs, SiBBlInGS allows for flexible variations in temporal traces within and between states, capturing the complete spectrum of activity variability while maintaining a shared basis of BBs for interpretability. Additionally, as shown in Figure 4E, the within-state temporal trace correlations are larger than between-states temporal correlations of corresponding ensembles, indicating reduced within state compared to between states variability.

Reviewer comment (Reproducibility):
Can the authors explain more on how SiBBlInGS handles missing samples and varied sampling rates in multi-dimensional data? How does it ensure that the discovered building blocks are robust to these variations?

Author response:
**Handling missing samples and varied sampling rates:**

We appreciate the reviewer's question regarding the treatment of missing samples and varied sampling rates. They are addressed in slightly different dimensions:

*Varying Sampling Rates:*

SiBBlInGS adeptly integrates data from trials or states, even when acquired from machines with varying sampling rates—a prevalent challenge in real-world data analysis. The model's channel similarity graph is constructed based on per-trial similarities, ensuring that cross-trial non-simultaneous observations or cross-trial observations with varied rates pose no hindrance. 

In its supervised option, SiBBlInGS uses the state labels to build the state similarity graph ($P$), thus independent from the sampling rate of different trials. 

In the inference phase, regularization between trials is exclusively applied to the BBs, which are not time-dependent and, consequently, unaffected by sampling rates. In contrast, the optimization for temporal traces (Equation 2) does not integrate cross-trial similarities, thereby accommodating variations in sampling rates.


*Missing Data:*

(1) When encountering missing samples, SiBBlInGS can simply exclude the missing points during the graph calculation process, and (2) it omits their impact in the data fidelity term during the optimization phase. 

In particular, let $\Pi$ be a projection operator that receives two matrices of the same dimension as inputs—a ''value" matrix (X) and a mask matrix (Y), such that if $Y_{ij} = 0$ , then $\Pi {(X,Y)}_{i,j} = 0$, 

and $\forall i,j$ s.t. $Y_{ij} \neq 0$, then 

$\Pi(X,Y)_{i,j}$

 = $ {X}_{ij}$.

Consider the above projection applied to the data fidelity term during the optimization (e.g., Equation 2), before applying the Frobenius norm: $|| \Pi (Y_m^d - A^d \phi^T , Y_m^d) ||_F^2$—specifically, this description ensures that the error in the data fidelity term does not take into account missing entries, using the mask from the observations $Y_m^d$.


We also note that we empirically showed on the synthetic data that the model is robust to missing samples, noise, and initializations (Figures 2 and 6).

Reviewer comment (Novelty):
What advantages does graph-based dictionary learning approach offer different from other dictionary learning approaches?

Author response:
**Advantages of graph-based dictionary learning:**

As detailed in the paragraph following Equation 1 (starting with 'The weighted evolving regularization terms for each state...'), the graph-enhanced re-weighted $\ell_1$ regularization differs from standard $\ell_1$ in that it incorporates temporal-similarity-driven weights that influence the sparsity of the BBs. This approach effectively reshapes the sparsity pattern of the BBs, ensuring not only sparsity but also grouping together channels with similar activity based on the channel graph $H$ that is integrated into the regularization. Additionally, the state graph $P$ also plays an important role by imposing an extra constraint, this time on cross-state BBs' similarity, making it proportionate to the similarity of the respective states. This results in an inference that is better aligned with the spatio-temporal information within the data compared to classical dictionary learning.